Observable Bloghttps://observablehq.com/blog/feed.rss Resources and insights on data visualization, dashboards, data apps, and more.Sat, 13 Jun 2026 22:23:43 GMThttps://validator.w3.org/feed/docs/rss2.htmlhttps://github.com/jpmonette/feedenCopyright Observable 2026https://observablehq.com/blog/from-raw-data-to-data-storytelling-framework-for-structuring-data-analysis https://observablehq.com/blog/from-raw-data-to-data-storytelling-framework-for-structuring-data-analysisFri, 03 Apr 2026 08:00:00 GMTWe’ve all been there — a new data analysis project lands on your desk, with a vague ask and a messy dataset. So you sort some columns, maybe add and remove a filter, and build some quick charts. But no clear pattern or trend stands out. After toiling away on the analysis for far too long, you emerge with a dashboard that tells you…not much. Your stakeholders are unsatisfied, and in the meantime more requests have piled up. Without a clear structure for [data analysis](/content/learn/what-is-data-analysis/index.html), it’s easy to find yourself in the weeds of an ambiguous project, without much to show for it. In this post, we outline a practical framework to efficiently move from vague requests to clear, useful insights. ## Refine the question A common mistake analysts make is diving directly into the data without fully understanding what’s being asked — and why. When that happens, the resulting analysis often showcases whatever an analyst can quickly see in the dataset, rather than answering a focused, decision-relevant question. But stakeholders often don’t know what kinds of questions they can ask of the data, or what would be most useful to see. Analysts should go beyond merely acting as order-takers, and take a larger role in helping stakeholders [define better questions from the get-go](/content/blog/role-of-inquiry-in-ai-data-analysis/index.html). This will be increasingly important as stakeholders conduct their own analysis using self-serve tools or AI. By working with stakeholders closely, data analysts can help to identify the most valuable questions, project goals, and scope. Consider asking clarifying questions when new requests come in, such as: - What are you trying to solve, and why does it matter right now? - Who is this analysis for, and how will they use it? - What decisions will this inform, and what would change based on this analysis? - What would be the most useful deliverable — is it really a dashboard, or would a simple chart, table, or summary fulfill this need? - Is this request appropriately scoped, or can we refine the question to focus on what’s most important? More time spent refining a request upfront can help avoid sprawling, undirected analyses that go unused because they don’t fill a specific need. Working closely with stakeholders is a core tenet of [collaborative analytics](/content/blog/collaborative-analytics/index.html), and it consistently leads to more insightful deliverables with fewer extraneous tasks and back-and-forths. ## Find answers in your data Once you’ve understood the request and its context, it’s time to dig in. This is the exploratory phase of data work, where you’ll discover what’s possible to learn and visualize — and what’s not. ### Explore and understand your data Take the time to [explore, profile, and understand your dataset](/content/blog/data-exploration-checklist/index.html). What’s the general shape of the data? How was it collected? What kind of data is contained in each row and column? Are there any missing values or duplicates? How are null values encoded? Are there mistakes or inconsistencies in the data that require correction before further analysis? This initial phase of [exploratory data analysis](/content/blog/three-ways-to-use-ai-for-exploratory-data-analysis/index.html) can often reveal patterns, outliers, and anomalies that can point you toward a deeper inquiry. By using summary statistics and other quality checks, you can start to map out what your analysis — and the resulting insights — will look like. [Observable Notebooks](/content/platform/notebooks/index.html) can be a great resource for this kind of exploratory work: they make it easy to inspect data, whip up prototype charts, examine different slices of data, and add notes and annotations as you conduct your analysis. You can also [fork](/content/documentation/notebooks/forking/index.html) another creator’s notebook to remix a visualization with your own data. ### Wrangle your data With a clearer picture of the data, you’re ready to get it into shape. [Data wrangling](/content/learn/what-is-data-wrangling/index.html) is the process of cleaning and formatting your data for further use in data analysis and visualization. The exact operations are specific to each data set, but often involve one or more of the following: - Joining tables across a relational database based on matching keys - Filtering out extraneous values or outliers to focus on the most important data - Pivoting or reshaping data to match the format your tooling expects - Deriving new values to standardize, normalize or convert for greater interpretability - Recasting variable types to avoid downstream errors in analysis - Handling missing or null values by exploring patterns of missingness After each of these transformations, take a step back and [pressure test your work](/content/blog/4-ways-to-avoid-common-gotchas-in-data-analysis/index.html). Did the transformation have the intended effect? Unchecked transformations can produce misleading results, so it’s important to inspect your data after each intermediate operation. [AI tools for data analysis](/content/blog/how-ai-is-accelerating-data-analysis-and-bi-work/index.html) can be particularly helpful during this phase, and accelerate repeatable tasks like data cleaning, wrangling, and code generation. Used thoughtfully, these tools can allow analysts to focus on the more human aspects of their role, such as asking better questions, testing new visualizations, and distilling analysis into clearer insights. But, we also know that AI occasionally hallucinates and makes mistakes, so analysts should closely inspect any AI outputs. ### Analyze and iterate to uncover insights With a clean dataset in hand, it’s time to start uncovering insights. This might mean calculating summary statistics, describing distributions and trends, modeling, and finding defined company metrics. Alongside the analysis, start building your data visualization. At this stage, clarity and speed matters more than polish. [Select several appropriate chart types](/content/blog/right-chart-for-data/index.html) for your data to compare different ways of looking at the data, and iterate to quickly identify the clearest and most useful story for your analysis. Throughout this process, make sure you’re keeping an eye on the north star: answering the refined question. This can help rein in scope creep and keep your analysis focused and useful. Here are a few guiding questions to ask as you carve out your story: - What findings most directly answer the stakeholder’s questions? - What kind of visualization would be most helpful? If multiple visualizations are needed, do they work together? - Is the scope of the deliverable in line with the question? ## Storyboard your findings with visualizations Once you’ve uncovered something in the data that most clearly addresses your stakeholder’s question, the next task may be the most difficult: deciding which pieces of your analysis will be included in the final story. Start by asking what essential knowledge someone should take away from the analysis. Not everything you find belongs in the final deliverable, and you’ll need to leave some work on the cutting room floor. Including too much information can overwhelm stakeholders and leave core insights buried in noise. The most useful analysis doesn’t aim to include all the data, but includes just enough that the audience understands what happened, why it matters, and what to do next. The story should stay grounded in the data, and include enough context or historical data for the audience to understand its significance. An analyst may be deeply familiar with the dataset, but your audience may not be. Without context, even a strong chart can be misinterpreted or underappreciated. It’s also worth thinking through the scale of the deliverable. The right format for a deliverable depends on the question, the stakeholder, and how it will be used. Sometimes stakeholders automatically request a dashboard, when in reality a number in context or small table is all they need. Matching the scope to the request can help make the analysis feel right-sized and complete, instead of either insufficient or overwhelming. ## Fine tune to meet stakeholders where they are So, you’ve got your analysis and data visualizations in hand. The final step is all about polishing how you present your analysis so it’s easy for stakeholders to understand and act upon. As you fine tune how you communicate your analysis, focus on distilling it down to its essential takeaways. To avoid any misinterpretation, try to use plain language, instead of jargon. Don’t leave stakeholders to draw their own conclusions — state what you learned, what it means, and proposed next steps in a clear and accessible way. Avoid [crowded dashboards](/content/blog/seven-ways-design-better-dashboards/index.html) or noisy charts, and resist the urge to share every single detail you’ve found. Adding some clarifying elements to data visualizations can also help stakeholders interpret your analysis and direct their attention to what matters. Here are some effective techniques for building clearer data visualizations: - Sort axes with intention: Using a meaningful sort can help viewers to quickly spot trends and compare values, especially across multiple groups or categories - Highlight the most important elements: Drawing attention to the most important elements of a chart can improve the interpretability of noisy or dense data - [Annotate important data points](/content/blog/five-techniques-to-improve-chart-annotations/index.html): Including more details around inflection points, events, patterns, or shifts in the data can help stakeholders by adding additional context - Give your charts actionable titles: Underscoring insights in the chart’s most important real estate can help stakeholders understand what they need to know, instead of burying the takeaway in a chart description or presentation talk track Also, think carefully about how and where your data analysis will be consumed. A chart in a dashboard may need a different level of explanation than a chart in a presentation or written report. An executive stakeholder skimming a weekly update may need the conclusion immediately, while a team reviewing a deeper analysis may want more detail on methodology and caveats. Will someone read this on mobile? Will it be updated in real time? Matching the format and detail to the audience and medium is part of what makes communication effective. ## Conclusion By taking the time to clarify the ask, explore and prepare the data, storyboard your findings, and fine tune to meet your audience where they are, you can produce analyses that are both more insightful and more useful. New tools like AI are leading to big changes across the data analysis industry, but the most valuable part of analysis will always come from the humans doing the work: by asking better questions, interpreting the data in context, and turning patterns into a story someone can understand and act on. By following this framework, you can chart a path from raw data to clearer data visualizations and analyses. \]\]>https://observablehq.com/blog/role-of-inquiry-in-ai-data-analysis https://observablehq.com/blog/role-of-inquiry-in-ai-data-analysisMon, 30 Mar 2026 00:00:00 GMTHow data analysts spend their time day-to-day is changing as they incorporate AI into their workflows. Tasks that used to take days or weeks, such as profiling a new dataset, cleaning messy data, or generating a prototype visualization, can now happen in minutes. The [promise of AI for data](/content/blog/how-ai-is-accelerating-data-analysis-and-bi-work/index.html) is that it can free analysts from rote computational tasks so they can focus on those that benefit most from human judgment: interpreting results, stress-testing assumptions, figuring out what a finding means for the business. However, there’s been little discussion of the role of human judgment _before_ AI enters the chat. AI has transformed analysis. It hasn’t yet done the same for the inquiry needed before analysis is executed — deciding what question is worth asking, defining your terms precisely, and understanding what you'd do differently depending on what the data says. Trusted, impactful data analyses only happen when teams ask the right questions. In late 2025, [only 10% of data practitioners](/content/blog/state-of-bi-and-data-analytics-2026/index.html) reported being confident in AI-generated insights. While that number has likely shifted, there’s still a lack of trust in the results of AI-powered analysis. What happens when AI produces technically correct answers to the wrong questions? The query ran cleanly, the chart looked right, the numbers added up — but the analysis still missed the point, because the question wasn't quite right to begin with. ## Good analysis starts with good questions Good analysis starts with good questions, which is why the best analysts take time to deeply understand their data and start analysis from a place of clarity. [The data analysis process is well-defined](/content/learn/what-is-data-analysis/index.html), but inquiry is often more fluid and less formal. When data analysis was slow and expensive, questions and hypotheses were naturally pressure tested. Analyses passed through multiple hands before reaching stakeholders and bad questions or assumptions got caught in review and refined through collaboration. The delay between asking and answering created natural space for reconsideration. AI makes it easy to bypass that space. Additionally, as AI democratizes access to data, stakeholders across the organization are now asking questions of data directly. While this gives stakeholders the ability to be more data-driven in their daily work, it also means more questions are being asked by people with less context to effectively evaluate the answers, uncertainty, and limitations. ## How to improve inquiry Now that AI has accelerated the execution part of data analysis, it’s important for analysts, and really anyone asking questions about their data, to invest in solid inquiry before simply firing off a bunch of prompts. Inquiry isn't just intuition. It's a learnable practice with identifiable components. Before any significant analysis begins, three questions are worth asking: If the data proves me wrong, will I know it? If the answer comes back differently than I expect, will anything change? And can everyone who reads this result agree on what the key terms mean? If the answer to any of those is no, the question isn't ready yet. **Is this question falsifiable?** A falsifiable question is one where the data could, in principle, tell you you're wrong. The opposite, a question structured to confirm what you already believe, isn't really analysis. It's validation. This shows up more than most analysts would like to admit: "Can you show me the impact of our recent campaign?" assumes impact exists and asks only for a narrative to explain it. "Did our recent campaign have an impact?" is a different question entirely. AI is particularly susceptible here to these nuances. Ask it to show you impact and it will find it, whether or not it's real. **Is this question decision-relevant?** A decision-relevant question is one where a different answer would produce a different action. If the data shows the opposite of what you expect, would anything change: a budget, a product decision, a strategy? If not, the question probably isn't ready yet. Because AI makes analysis so cheap to produce, the volume of low-decision-relevance work is only going to increase. Decision relevance is the filter that keeps inquiry focused on what actually matters. **Are the key terms defined precisely?** Every key term in a question should have a single, agreed-upon definition that will be calculated and interpreted consistently. "Active users," "revenue," "churn" mean different things to different teams, and AI will resolve that ambiguity silently by using whatever definition is most available in the data. These three criteria matter individually. But their organizational implications are what make them urgent, especially when an AI agent can operationalize a poorly defined metric across an entire organization in seconds. ## What this means for data teams Data teams are uniquely positioned to help operationalize good inquiry within their organizations. Their first priority should be to model good inquiry practices by making inquiry an explicit part of the data analysis process. Documenting good inquiry habits and sharing these practices internally can help create a repeatable standard for both analysts and stakeholders. The second priority is to help stakeholders improve their own inquiry and analysis skills. Self-serve analytics exposed more stakeholders to data, and AI has widened the aperture even further. As more people across the organization gain direct access to data through AI tools, data teams have an opportunity to raise the quality of the questions being asked organization-wide. The key is that this is accomplished through collaboration and shared learning — not gatekeeping. Lastly, both data analysts and stakeholders need to develop rigor around verifying AI’s output. Good inquiry is an important input for quality analysis, but it’s not the only factor at play. AI makes mistakes. An aesthetically attractive chart isn't the same as a correct one. Anyone using AI for analysis needs to validate the output for accuracy. ## Conclusion AI is only going to get faster and more capable. The volume of data that AI analyzes will increase, and the bar for producing a polished-looking result will keep dropping. As it does, the importance of asking good questions that yield trusted, decision-relevant answers will only increase. While there's a lot we don't know about the future of AI-powered data analysis, what is clear is that data teams that invest in building a practice and culture of rigorous inquiry now will be better set up to succeed in the future. \]\]>https://observablehq.com/blog/the-medium-is-the-problem https://observablehq.com/blog/the-medium-is-the-problemWed, 25 Mar 2026 00:00:00 GMTAI can write code. But neither you nor it understands what it's written. The reason has less to do with the code itself than with the medium where it runs. Since the earliest days of programming, software development has followed a tight, repeatable, legible loop: write code, run code, inspect the result. Every major advance in tooling has fundamentally been about making that loop faster and more transparent. Implicit within that process, so foundational as to be almost invisible, was the human engine starting it, writing it, and making it go. In modern AI development, vibey or otherwise, we no longer start by writing code. Humans start by describing intent and then watch as the system generates an implementation. We know that the result is often something far more capable, and far more complex, than anything a person could have produced themselves. Less obvious is that it's also more complex than a person would produce if they could. The code looks plausible. It runs. But with software, the question isn't only whether code works — it's whether it's right. And to answer that, as a developer you now have to read through logic you didn't write, trace variables you don't recognize, and rerun unfamiliar pieces of a system you didn't design, trying to answer questions as basic as: what is this doing, and why? Software is valuable when it is correct, trusted, and extensible. All three depend on comprehension. AI increases the volume and velocity of code, but not its comprehension. Very often, it does the opposite. This is the gap. The solution isn’t in the — it’s in the environment where code runs. Today, most workflows treat execution as secondary — something that happens after the real work of generation is done. Code is produced in one place and run in another. Results come back, but intermediate state stays hidden. Interestingly, insidiously, the system behaves like a black box even when the code is fully visible. We have dramatically improved "write" without corresponding improvements to "run" and "inspect." When an agent can generate a thousand lines in thirty seconds, a once-manageable comprehension gap now opens at a scale no human review process can keep up with. What this moment demands is an environment that can keep pace with generation — one that makes execution visible, exposes state as it changes, and supports rapid iteration between human and machine not after the fact but as the system builds. The loop that defined software development for decades — write, run, inspect — has to be restored, but at the speed and scale of machines rather than humans. Without that, we are simply moving faster toward the same opacity, generating more and understanding less. Until then, AI will be an extraordinary way to produce code, but not extraordinary software. \]\]>https://observablehq.com/blog/beyond-bar-charts-a-guide-to-advanced-visualizations-in-observable-plot https://observablehq.com/blog/beyond-bar-charts-a-guide-to-advanced-visualizations-in-observable-plotMon, 23 Mar 2026 08:00:00 GMTMost data analysts cut their teeth on a familiar set of chart types when learning data visualization skills — among them, the bar chart, the line chart, the pie chart, and the scatter plot. These standard charts are widely understood, easy to build, and dependable for many common analyses. They come pre-built in most business intelligence or AI for analysis tools, and are often the first chart types that data teams use for exploring trends and patterns in a dataset. So it follows that when a request comes across an analyst’s desk, they’ll often reach for one of these standard charts. But in some cases, they may not be the best fit for the data, and a more sophisticated chart could add more value or uncover more insight. For example: - A bar chart can compare aggregate values by category, but it may hide patterns within those observations. - A line chart can show change over time, but noise and overlapping lines might make it difficult to decipher. - A pie chart can show parts of a whole, but it can make proportions hard to compare. - A scatter plot can reveal relationships between two variables, but dense observations can obscure the broader pattern. In this post, we take a closer look at some advanced chart types that you should consider adding to your data visualization toolkit. You’ll learn when to consider these charts, and we include [Observable Plot](/content/plot/index.html) code that you can use to quickly spin up working prototypes. The charts below do not fit every situation and are not one-for-one replacements for standard chart types. When [choosing a chart type](/content/blog/right-chart-for-data/index.html), consider the structure of the data you’re trying to visualize. Second, look for a chart that clearly conveys an insight or pattern without adding unnecessary clutter or noise. Finally, select a chart that provides enough detail to give that insight useful context. Read on to learn about 6 effective, advanced data visualizations that are easy to build with Plot: ## Bar chart alternatives ### Barcode chart A [barcode chart](/content/@observablehq/plot-barcode/index.html) plots individual values as thin marks along a single axis. Instead of binning values into bars or reducing them to a summary statistic, it lets readers see the distribution of values so they can observe clusters, gaps, outliers, and spread. Barcode charts are especially useful when the shape of the data matters more than a summary count. They work well for exploratory analysis and for datasets such as page load times, prices, durations, or event timing. If it’s valuable to see the distribution of individual observations, a barcode chart may be a better fit than a bar chart or histogram. This chart type works well with smaller datasets, because plotting too many values can make a barcode chart visually dense. You can build a barcode chart in Observable Plot using [tick marks](/content/plot/marks/tick/index.html). ``` Plot.plot({ x: {domain: [1.2, 2.2]}, color: {legend: true}, marginLeft: 100, marks: [\ Plot.ruleX([0]),\ Plot.tickX(olympians, {x: "height", y: "sport", opacity: 0.2, strokeWidth: 3, stroke: "sex"})\ ] }) ``` ## Line chart alternatives ### Streamgraph A [streamgraph](/content/@observablehq/plot-stack-offset/index.html) is a type of a stacked area chart that shows how categories change over time compared to a central baseline. Readers can easily understand the direction and magnitude of change over time for different groups of data by following the thickness of each stacked layer. Streamgraphs are often used to display time-series data, such as website traffic or sales. While line or bar charts can show individual values more precisely, a streamgraph may be a more expressive choice for visualizing the trend and flow of categorical data through its smoothed curves. You can build streamgraphs using Observable Plot using stacked areas with a [wiggle or center offset](/content/plot/transforms/stack#stack-options/index.html). Plot also offers other anchoring options to change stacked layer baselines, including null and normalize. ``` Plot.plot({ y: { grid: true, label: "Unemployed" }, marginLeft: 50, color: { legend: true, columns: 4, width: 600 }, marks: [\ Plot.areaY(industries, {y: "unemployed", x: "date", fill: "industry", interval: "month", offset: "wiggle"\ })\ ] }) ``` ### Ridgeline plot [Ridgeline plots](/content/@observablehq/plot-ridgeline/index.html) display multiple distributions as a series or overlapping density curves. This helps readers compare the shape and spread of observations across grouped distributions. A ridgeline plot can be a more compact or elegant choice instead of a series of histograms for revealing broader patterns or general shape of distributions. For example, radio signals, seasonal temperature patterns, or street traffic data are well-served by ridgeline plots. However, the shape of one distribution may occlude another, so a ridgeline may not be the best chart for precise comparisons across groups. Consider a [horizon chart](/content/@observablehq/plot-horizon/index.html) for a similar compressed view that avoids overlap. The snippet below creates a ridgeline plot in Observable Plot using the [line](/content/plot/marks/line/index.html) and [area marks](/content/plot/marks/area/index.html) in combination. ``` Plot.plot({ y: {axis: null, range: [50, 2]}, height: 500, marginLeft: 150, marks: [\ Plot.areaY(industries, {x: "date", y: "unemployed", z: "industry", fy: "industry", fill: "#ccc"}),\ Plot.lineY(industries, {x: "date", y: "unemployed", z: "industry", fy: "industry", strokeWidth: 1})\ ] }) ``` ## Pie chart alternatives ### Waffle chart A [waffle chart](/content/@observablehq/plot-stacked-waffles/index.html) shows parts of a whole using a grid of equally sized units. Instead of comparing slices in a pie chart, or sections of a stacked bar chart, the reader can easily compare exact quantities and proportions. This can be especially useful when you want to show the composition of a dataset in a way that feels intuitive at a glance. [Survey results](/content/@observablehq/plot-survey-waffle/index.html), market share, and categorical breakdowns are all strong use cases for waffle charts. One thing to note is that waffle charts are not ideal for precise comparisons across a large number of groups, or when the total quantity does not break cleanly into simple units. But if you’re comparing a small number of categories, and want to help the reader compare exact amount or proportion, waffle charts may be a better fit than a pie or bar chart. You can build a waffle chart in Observable Plot using the [waffle mark](/content/plot/marks/waffle/index.html). Like a bar chart, a waffle chart can be stacked using the [stack transform](/content/plot/transforms/stack/index.html). ``` Plot.waffleY(penguins, Plot.groupZ({y: "count"}, {fill: "species", sort: "sex", fx: "island", unit: 1})).plot({color: {legend: true}}) ``` ## Scatter plot alternatives ### Hexbin heatmap A [hexbin heatmap](/content/@observablehq/plot-olympians-hexbin/index.html) groups continuous datasets into discrete hexagonal bins, and encodes the summary statistics (such as count, average, or proportion) through the hexagon’s color. This makes it a valuable tool for visualizing dense quantitative or temporal data in two dimensions where a scatter plot would be too crowded to parse. Hexbin heatmaps aggregate the data, so you lose visibility into individual observations. Bin size also matters: if the hexagons are too large, it can be difficult to uncover a trend or pattern. If the bins are too small, the chart can become noisy. But if your goal is to give readers a sense of the overall density of the data, hexbin heatmaps are a good candidate. You can build hexbin heatmaps in Observable Plot using [hexbin transforms](/content/plot/transforms/hexbin#hexbin-transform/index.html). ``` Plot.plot({ color: { scheme: "ylgnbu" }, marks: [\ Plot.hexagon(olympians, Plot.hexbin({fill: "sum"}, {x: "weight", y: "height"}))\ ] }) ``` ### Density chart A [density chart](/content/@observablehq/plot-olympians-density/index.html) visualizes concentrations of data points instead of individual observations. Like the hexbin heatmap, density charts are useful for crowded, dense datasets, or when you want to illustrate clustering, hot spots, or distributions in two dimensions. Density charts work well for continuous variables where the overall pattern matters more than any single observation. When a scatter plot would appear crowded or noisy, a density chart might be a better fit for the data. Observable Plot’s [density mark](/content/plot/marks/density#density-mark/index.html) makes it easy to build several variations on this chart type. ``` Plot.plot({ marks: [\ Plot.density(olympians, {x: "weight", y: "height", stroke: "sex"}),\ Plot.dot(olympians, {x: "weight", y: "height", r: .1, stroke: "sex", opacity: .5})\ ] }) ``` Plot also supports density visualization to visualize geospatial data. Here’s an example of a visualization using the density mark to [plot Walmart stores](/content/@observablehq/plot-walmart-density/index.html) on an equal-area projection of the U.S. ## Conclusion Bar charts, line charts, pie charts, and scatter plots remain essential tools in data visualization. But, it’s worth building out your data visualization toolkit and exploring more advanced options that may unlock unseen insights, or help you visualize different aspects of your data. There are many chart types to explore, along with other data visualization elements such as [chart annotations](/content/blog/five-techniques-to-improve-chart-annotations/index.html) that can direct a reader’s attention, and [animations](/content/blog/effective-animation/index.html) that can help a reader follow along with changes over time. As you experiment with new ways to visualize data, explore the [Observable Plot gallery](/content/@observablehq/plot-gallery/index.html) for examples, templates, and ideas you can adapt for your own work. \]\]>https://observablehq.com/blog/what-data-teams-can-learn-from-journalists-about-data-visualization https://observablehq.com/blog/what-data-teams-can-learn-from-journalists-about-data-visualizationWed, 18 Mar 2026 08:00:00 GMTData journalists are tasked with turning messy and complex data into digestible visualizations and engaging stories that can capture attention in the daily scrum of online media. Data teams may serve a different audience, but they face a similar challenge: turning analysis into something a stakeholder can quickly understand and act on. By adopting some of the practices of data journalism, data teams can deliver more effective analyses, and build visualizations that are clearer and more useful to their stakeholders. Below, we highlight practical strategies from leading newsrooms that data teams can use in their day-to-day work, based on r [ecent conversations with data journalists from Axios and Reuters](/content/blog/webinar-recap-storytelling-with-graphics-from-raw-data-to-reader-impact/index.html). ## Consider your audience’s knowledge and goals When diving into a project, data teams may make the mistake of assuming stakeholders will understand their deliverables entirely. Newsrooms don’t have this luxury, and must fight for each click and view. Data teams should take a step back to think critically about their audience before diving into analyses. What might they understand (or not understand) about the data? What context would they need to interpret any given insight? What goals do they have, and what challenges are they trying to solve? These questions have a direct impact on the final product, including what form (or forms) a deliverable will take, and how much data is presented. On our recent webinar, Axios Associate Editor Kavya Beheraj highlighted these important questions as foundational for data work: > At my past job, I had a lot of experience making dashboards that I'm not sure anyone actually used or saw. And I think part of the challenge there is that we're going in assuming that we need a dashboard and that all of the data points that we want to visualize matter. But I think there are more fundamental questions, like, why do we need to visualize all of this in this way? Who is going to be using it, and how? And how is it going to be maintained? Those are principles from data journalism that apply anywhere. Anytime you want to do some sort of visual communication or chart, it is helpful to understand who you are trying to help, to influence, to inform. Kavya Beheraj, Associate Editor, Axios ## Take the time to deeply explore and understand your data Real-world datasets are rarely dashboard-ready, and may contain missing values, outliers, or mismatched nulls. [Data exploration](/content/learn/what-is-data-exploration/index.html) is a critical step in the analysis process, where analysts can familiarize themselves with the data, how it was collected, and start building hypotheses around what insights can be gleaned from it. During our recent webinar, Ben Welsh, News Applications Editor at Reuters, pointed out that it's important to focus on the fundamentals and ask even the most basic questions of datasets — for example, what does any given row represent? Assuming something as fundamental as all rows equating to a single observation can lead teams down the wrong path, as he discovered while working with tree planting data from the [City of Chicago](https://data.cityofchicago.org/): > There was a column with a very cryptic name — we didn't know what it was and didn't think too much about it — but it was actually the number of trees that had been planted in that row. Each row was not a tree, which is what we were working under the assumption of. Each row was a work order to plant one-to-many trees. So our initial analysis was fundamentally wrong because we didn't understand everything we needed to know about the data. Just slowing down and forcing yourself to walk through that is crucial. What is actually being shown? Maybe it's multiple things at once, maybe there's multiple data sets mixed together that need to be teased apart. Ben Welsh, News Applications Editor, Reuters By thinking more like journalists, analysts can interrogate their datasets, and start to better understand it and what stories it can tell: Why does this dataset exist? How were these observations collected? How is it already used or manipulated? Who needs this, and what will they do with it? From there, as Kavya says, “let the story steer the ship.” Instead of simply jumping into a dashboard builder based on preconceived notions about the data, the data and the insights uncovered within it should inform the analysis and how it’s [communicated](/content/learn/what-is-data-analysis#5-interpretation-and-communication/index.html). ## Collaborate for stronger visualizations and deeper insights Effective collaboration can streamline processes and improve the end result of data analysis, and nowhere is that more apparent than the fast-paced world of online media. Data teams can work more efficiently by practicing [collaborative analytics](/content/blog/collaborative-analytics/index.html), and bringing stakeholders into the data analysis process directly. Data teams are often most knowledgeable about an organization’s data; stakeholders often have the clearest understanding of an organization’s operations and challenges. Bringing analysts and stakeholders together during data work can reduce tedious feedback cycles, speed up project timelines, and illuminate insights that would otherwise be missed. > When you find yourself in a collaborative environment, you probably shouldn't be the best designer, the best developer, the best data journalist, all these things at once. Learn to be comfortable with putting your ego aside, and let the people who shine at something, shine. Not only will that make for a smoother and better process — and probably better product in the end — it also will allow you to grow into your preferred specialty. Kavya Beheraj, Associate Editor, Axios ## Personalize visualizations to improve engagement Data journalists often are tasked with telling stories around large, complex, and abstract trends. Localizing these stories is one way to drive audience engagement. During our conversation, Kavya highlighted one example of a dashboard that allowed readers to see the effects of wildfire smoke in their communities. “We wanted a way to take this really large national story and make it relevant in some way to folks — how it compares to the average, whether what you're experiencing is healthy or unhealthy according to the standards set by the government… The stories got \[picked up\] a lot, because we took the time to localize it.” The team at Axios built a [dashboard](https://www.axios.com/us-climate-change-air-quality-canada-wildfire-season) that helped readers understand how wildfire smoke was impacting their community's air quality and pollution exposure. Data teams can adopt similar practices by delivering more personalized insights, and making reporting more granular. For example, instead of creating a single global sales dashboard, their deliverable could include simple interactivity so sales leaders can easily customize reports to their region or team’s specific metrics. ## Design charts for interpretability Data visualizations should not need a presenter to explain what matters. When telling stories with data, journalists [choose the chart](/content/blog/right-chart-for-data/index.html) that best fits the story and the data, then make careful design decisions so the result is engaging and interpretable. Sometimes that means using a simple chart that clearly communicates a single important point. It could also mean building a more immersive data story that reveals something a standard chart cannot. Data teams should aim to create visualizations that provide enough context to be understood on their own. When was the data last refreshed? What should the viewer compare it against? Is the result positive or negative? Expected, or unusual? That context can come from annotations, benchmarks, or historical data that shows how recent observations compare with past performance. As organizations move beyond the dashboard paradigm, stakeholders may expect or look for data visualizations elsewhere, such as in Slack or Microsoft Teams, inside data apps, in documents, or on mobile devices. Data analysts should consider where their charts will be viewed, and adapt their charts to [better utilize constrained spaces](/content/blog/big-insights-small-spaces/index.html). A chart that can stand on its own across a wide variety of surfaces is more likely to be understood, reused, and acted on. ## Use AI to automate programmable work and increase understanding Data journalists are often early adopters of new technology, in part because their work constantly exposes them to messy, unpredictable datasets and the tools needed to make sense of them. As Ben Welsh put it in an [earlier conversation](/content/blog/ben-welsh#innovating-in-a-changing-industry/index.html), "We’re constantly encountering all the different weird things that can happen with data. Being the nerd caught in the middle of that mucky data processing is something that most people who work in data journalism end up liking. But it’s also why any technology that makes it easier for us is really exciting, and is why we tend to be rabid early adopters of weird new data tools." AI is one of the latest examples. It has shown real value in automating repetitive work, which gives data journalists more time for the part of the job that requires human ingenuity: interpreting data and communicating what it means. In business settings, that same principle applies to recurring data prep and cleaning, dashboard maintenance, and routine QA. By automating these manual, less creative tasks, analysts can spend more time asking questions of their data to uncover the insights that move the needle for their organization. AI can also help audiences interpret data and stories. As Ben pointed out, “It’s going to increase the accessibility of the type of work we do, by bringing more people into the conversation.” Likewise, data teams can deploy AI tools that enable stakeholders to explore data on their own, without requiring advanced SQL or coding skills. ## Conclusion Data journalists work in an environment where attention is at a premium. They understand that data analysis is only effective when it’s communicated and understood. These processes and data visualization principles, honed in the newsroom, can provide a useful model for analysts who are looking to tell better stories with their data. The result will be more insightful analysis and deeper engagement with stakeholders, who will be better able to understand the findings and use the work to make more informed decisions. \]\]>https://observablehq.com/blog/shifting-stakeholder-expecations-in-the-age-of-ai-powered-analytics https://observablehq.com/blog/shifting-stakeholder-expecations-in-the-age-of-ai-powered-analyticsWed, 04 Mar 2026 00:00:00 GMTFor years, dashboards have been the de facto deliverables for many data analysis projects. They certainly have their strengths: polished, long-lived artifacts are useful when tracking established metrics or exploring trends over time. But we also know that many dashboards start to [rot](/content/blog/five-strategies-fight-dashboard-rot/index.html) as soon as they’re shipped. Unless data teams dedicate ongoing resources to maintenance and continued development, dashboards can break or lag behind evolving company needs. Stakeholders stop returning to dashboards when they have to [leave their day-to-day tools and workflows](/content/blog/state-of-bi-and-data-analytics-2026#breaking-charts-out-of-the-bi-dashboard/index.html), track down the right link, and relearn how to interpret the dashboard content. Perhaps most importantly, stakeholders are realizing that answers to many questions that pop up in their day-to-day work often aren’t found in dashboards: they’re increasingly found in quick chats with teammates, or through self-serve discovery enabled by AI assistants. As a result, expectations about where and how stakeholders interact with data are changing. Here, we describe shifting user preferences and habits, and share how data teams can meet stakeholders where they are in today’s emerging AI-powered data landscape. ## **Users want to explore data independently in the flow of work** Until recently, it was standard practice to publish a dashboard and assume decision makers would return to it when they needed answers. But stakeholders today are unsatisfied with consuming prebuilt insights. They want to investigate ad hoc questions as they arise, without having to wait in their data team’s ticket queue. As Observable co-CEO Julio Avalos shared in our [recent webinar](/content/blog/webinar-recap-evolving-data-analysis-landscape/index.html), this is an evolution years in the making: > We've spent decades convincing business managers, executives, and even individual contributors throughout an organization that data is meaningful, and they therefore want to participate in finding insights. They don’t feel entirely comfortable offloading data work onto specialists within their organization — they want faster answers, and more self-serve access. Julio Avalos Observable Co-CEO With AI agents quickly democratizing data analysis, it’s not just _possible_ for more people with diverse skillsets to jump into data work — it’s becoming the norm. [AI agents are powerful for common tasks](/content/blog/how-ai-is-accelerating-data-analysis-and-bi-work#where-ai-is-helping-analysts-today/index.html) like summarizing trends, surfacing anomalies, translating natural language into queries, and drafting data visualizations. Boosted by AI, product managers, researchers, and even casual hobbyists now have the ability to independently investigate and answer questions. Just as important as _how_ stakeholders want to interact with data is _where_ that interaction happens. People want to easily access and work with data without jumping out to a separate BI platform, which slows momentum. Modern tools help users stay in flow by exploring metrics inline, adjusting assumptions in context, and sharing findings instantly without saving and emailing screenshots. Here’s Julio again: > People don't want to go and learn yet another tool. I think that's another changing expectation for stakeholders: they want to be met where they are. They want to be met in Slack. They want to be met in Teams. They want to be met wherever they're working, and they want to consume your information and participate in data work there. Julio Avalos Observable Co-CEO To summarize, stakeholders are moving away from dashboards as the go-to place to find insights in their data. Instead, they’re turning to AI agents that let them immediately dig into new questions as they pop up, without filing a ticket or breaking out of their existing workflows. ## **How data teams can navigate shifting stakeholder expectations and practices** Adapting to these evolving expectations doesn’t require sacrificing rigor. As AI-powered analytics becomes more widespread, data teams play a critical role in enabling this new way of working while safeguarding trusted, high-quality practices. Here are three practical ways data teams can meet stakeholders where they are, without compromising analytical standards: 1. **Model and teach reproducibility.** Reproducibility becomes even more critical as insights are generated in AI chats and shared across ephemeral channels, because it ensures accurate, trusted answers that hold up to scrutiny. When a metric appears in a thread, it should link back to a source of truth such as a version-controlled query, a documented definition, or a computational notebook with necessary code to recreate the results. By encouraging good practices and tooling for reproducible analysis, analysts can help users across teams produce verifiable results for more confident decision-making. 2. **Ship portable, digestible, and focused insights.** Rather than adding to sprawling dashboards, analysts should move toward delivering digestible insights and data visualizations that answer specific questions. A focused chart embedded in a ticket, thread, or shared document invites more engagement than one locked inside an unfamiliar BI workspace. 3. **Define clear standards around exploratory versus production-grade outputs.** In fast-moving conversations, it is tempting to immediately share whatever number an AI agent returns. Sometimes, that might be low-risk. Or, it can have serious business consequences if AI hallucinates or misinterprets your request. Analysts can help their teammates distinguish between quick exploratory analyses, and production-grade outputs that require more rigorous testing and validation. This empowers users to practice fast, AI-powered analytics when appropriate, and know when they need to ask for additional review. By modeling and teaching reproducible workflows, shipping digestible insights, and helping to differentiate between low-risk exploration and consequential analyses, data teams can meet stakeholder expectations for faster AI-powered discovery while promoting sound practices across their organization. ## **Conclusion** Dashboards aren’t disappearing, but they’re no longer the center of gravity for data work. Stakeholders expect to find insights in real time within their everyday tools, enabled by AI agents that democratize data exploration and analysis. In this evolving landscape, data teams can meet stakeholders where they are by delivering digestible, inspectable insights directly into their existing workflows, and advocating for practices that uphold analytical standards. Want to hear more about how AI is impacting the data analysis landscape, and how data teams can keep up? Watch the recording of our [recent webinar](https://www.youtube.com/watch?v=nzpF_j_GAmQ), or read the [recap blog post](/content/blog/webinar-recap-evolving-data-analysis-landscape/index.html) for highlights. \]\]>https://observablehq.com/blog/how-ai-is-accelerating-data-analysis-and-bi-work https://observablehq.com/blog/how-ai-is-accelerating-data-analysis-and-bi-workThu, 19 Feb 2026 08:00:00 GMTThe data analysis and BI industry is no stranger to change, but the speed and reach of advances in AI have caused major shifts to analytical workflows. AI is rapidly transitioning from an experimental tool to operationalized technology that is core to data teams’ daily work. As AI automates tasks throughout the [data analysis](/content/learn/what-is-data-analysis/index.html) lifecycle, analysts can refocus their effort on the things that require human ingenuity and expertise like asking better questions, interpreting data, and applying their judgement at scale. While AI is meaningfully improving data workflows in some areas, there are still areas where AI tools miss the mark. Below, we walk through how AI is meaningfully contributing to data team workflows, and where its promised capabilities don’t always align with today’s reality. We also share what we see as the biggest untapped potential of AI tools, and what data teams can do to make the most of this new technology. ## Where AI is helping analysts today ### Automating the “least human” parts of data work When we spoke to data analysts and BI professionals about the role of AI in their day-to-day work, many pointed to using AI for mundane tasks. Steps like initial [data profiling](/content/blog/three-ways-to-use-ai-for-exploratory-data-analysis#use-ai-for-fast-data-profiling/index.html), summarization, and cleaning are critical in the data analysis process, but typically aren’t how the average analyst would want to spend the majority of their time. So, it should come as no surprise that data teams reach for AI tools to automate these repetitive tasks that are invaluable for [avoiding thrash and common mistakes](/content/blog/4-ways-to-avoid-common-gotchas-in-data-analysis/index.html), but that don’t always require human creativity or judgement. AI can be a powerful tool for exploratory data analysis and wrangling, enabling analysts to quickly inspect datasets and identify issues to avoid costly errors down the line. As data teams adopt AI tools to automate mundane tasks, analysts can shift their resources and energy to more interesting aspects of data analysis like model evaluation and interpretation, or fine-tuning data visualizations. Here are example AI prompts that can help with your next data exploration: - Are there any null or missing values in this dataset? - Do missing values appear to follow a pattern? - Summarize the schema of this dataset in plain English. What does each table represent? ### Simplifying complexity and growing data skills Another area that AI shines is breaking down complex information into its component parts, which encourages learning and skill-building. This is particularly helpful with technical tasks like understanding and debugging code — especially code written by other developers or data scientists. Analysts’s resources are often stretched thin, with only the time and bandwidth to ship a final artifact. In the rush to move on to the next request, however, helpful documentation and explanatory materials are often left by the wayside. But expecting the insights to be self-evident to stakeholders can in fact devalue the analyst’s choices and work. AI tools can fill the gap by summarizing key insights and creating documentation to guide stakeholders as they explore dashboards and other deliverables. Finally, data is everywhere, and stakeholders need a baseline level of [data literacy](/content/learn/what-is-data-literacy/index.html) to keep up. AI can help here as well! These tools are upleveling data skills and knowledge organization-wide, giving cross-functional teams access to a personalized coach to support their data skill development that adapts to their level of expertise. ## Where AI falls short in data work ### Reproducibility and trustworthiness are still a major challenge When AI tools produce insights or analyses, it’s critical that the analyst can reproduce the output. We think of the challenge as akin to a reporter looking for confirmation on a story from a second source — if the AI uses the same data and prompt, but comes back with a different response, that’s an intolerable situation for organizations trying to make sense of their data. No analyst is going to want to put that report in front of stakeholders or clients and no stakeholder will feel confident acting upon it. By now, most people understand that AI tools can misinterpret requests, or lack context while providing a technically correct answer. Worse, major AI tools continue to hallucinate, or generate synthetic datasets to provide erroneous responses when it can’t find the appropriate tables in a database. The problem is that bad AI “doesn’t smell” — its outputs may look legitimate, while being entirely fictitious or flat out wrong. So, skepticism around using AI tools for mission-critical analytical work is entirely valid. Analysts must be able to trust the results of any tool they use. ### General-purpose LLMs weren’t designed for real-world challenges of data analysis In addition, general-purpose LLMs were not built specifically for data analysis workflows. They don’t automatically understand the nuances of a particular schema, the reason behind certain business operations, or edge cases presented by real-world data analysis work. They also don’t retain the rationale behind past business decisions. That creates a major gap for general-purpose models: why a metric was defined a certain way, why a filter was applied, what trade-offs were made. When prompted to generate visualizations, these general-purpose LLMs will typically produce only basic and non-interactive charts, potentially leaving insights on the table. ## AI’s untapped potential At the same time, these challenges present an opportunity for data teams to use these AI tools to create more efficient processes. For example, AI tools could help data teams avoid losing institutional memory and recreating past work by enshrining business decisions — and the reasoning behind those decisions — and corporate values directly into their analytical processes. Specialized tools, designed for the rigors of modern data analysis work, can augment analysts’ work through transparent, interpretable interfaces, and encourage deeper explorations. AI tools can deliver a more [collaborative analytics](/content/blog/collaborative-analytics/index.html) process and bridge the gap in data literacy between analysts and stakeholders, helping members of cross-functional teams become active participants in insight discovery. There’s also a big, often-overlooked gap here around data visualization. If a chart requires deep knowledge of underlying analysis to be useful, it can be a sign that the visualization (or the way it’s being shared) is missing the mark. Most organizations massively underinvest in expressive, clear data visualizations — even though better visuals can make data instantly more intelligible and interpretable. AI can step in here to level up analyst capabilities in delivering advanced, expressive visualizations that communicate insights clearly. ## Conclusion While AI excels at automating repetitive tasks and breaking down complexity, the human elements of data work — curiosity, judgment, contextual understanding, and the ability to ask the right questions — remain irreplaceable. Perhaps paradoxically, that makes human input and expertise increasingly critical as AI handles more of the mechanical work. For data teams navigating this transition, automating repetitive tasks is a proven use case for AI tools, including data profiling, basic exploration, documentation, and debugging, along with improving collaboration and data literacy organization-wide. At Observable, we believe AI as an accelerator for human capability and ingenuity. The data teams that thrive in this new era will be those that recognize AI as a powerful tool to augment data teams, not a replacement, freeing analysts to focus on what they do best: turning data into insights. \]\]>https://observablehq.com/blog/webinar-recap-evolving-data-analysis-landscape https://observablehq.com/blog/webinar-recap-evolving-data-analysis-landscapeThu, 12 Feb 2026 08:00:00 GMTThe BI and data analysis industry is reaching an inflection point, and data teams must navigate a moment of massive upheaval in their day-to-day work. LLMs and agentic analytics tools are rapidly advancing, and stakeholder expectations are shifting: they want answers faster, and they want to find insights themselves. In a [recent fireside chat](https://www.youtube.com/live/nzpF_j_GAmQ?si=uhe1nZvQbp2sViaO), Julio Avalos, Co-CEO of Observable, and Marisa Morby, our Director of Research, unpacked what these transformations mean for data teams and proposed ways analysts can stay at the forefront of this changing BI and analytics landscape. Their conversation touched on findings from our recent [State of BI and Analytics survey](/content/blog/state-of-bi-and-data-analytics-2026/index.html), and outlined where AI is having real impact, where it falls short, and how analysts can get ahead of changing stakeholder expectations. ## What are some of the broader changes across the BI and analytics industry? The data space continues to reckon with the now familiar challenges of technological evolution, rising user expectations, budget constraints, and resource limitations. But now, AI is significantly changing the landscape. Simultaneously, stakeholder expectations demand faster, self-serve answers. This combination is reshaping how data teams operate. In the webinar, Julio pointed out that data is everywhere — not just large data warehouses. It also shows up as small, personal data in consumer and internal apps, meaning people interact with data constantly, often without realizing it's "data work." Yet, exposure doesn't always lead to data fluency. While people may be more comfortable with charts, they aren't necessarily better at asking good questions. There’s also an issue of scale and added complexity. Organizations collect more data than ever, but many existing tools and workflows were designed for a different era. Stakeholders are breaking away from the paradigm of passively consuming charts and reports that defined corporate data work for decades. This shift creates human and technical uncertainty around what questions to ask, and how to trust data and results. Issues of provenance, credibility, and trustworthiness are also exacerbated by AI. As Julio described, this means we’re reaching a moment where human ingenuity matters more, and not less: > If AI takes an increasing role, the value of human insight — the "spark" — becomes paramount. Data analysis within businesses needs to become more like continuous, collaborative flows, demanding new tools and paradigms to support this reality. Julio Avalos, Co-CEO, Observable ## AI’s impact on data analyst workflows ### How is AI meaningfully helping analysts today? AI is helping analysts with the more mundane parts of the job, Julio observed, including [data profiling](/content/blog/three-ways-to-use-ai-for-exploratory-data-analysis/index.html), anomaly detection, quality checks, and other repetitive tasks that require a significant investment of time, and that don’t directly benefit from human creativity or judgment. By automating these rote tasks, analysts can spend more time adding value to their organization by doing the work they were hired for: finding and communicating useful insights in the data. Marisa shared that in research calls, analysts often echo that AI is particularly useful at automating the less creative or administrative functions of data work. She also pointed out that debugging complex SQL — especially SQL written by other developers — was another popular use case for AI among data analysts. AI also has the potential to improve data literacy, and to help users develop new skills and techniques. ### Where is there a gap between promised AI capabilities, and what AI can deliver? AI for data analysis is in its early stages, Julio said, and early generations of AI tools have not been built specifically with data analysis in mind. This has led to a few key gaps. Reproducibility is one such critical gap. If an insight cannot be reproduced using the same tool, data, and prompt, it cannot be trusted by the analyst or stakeholders. A second, related gap is verifiability. Much of today's AI operates as a black box. As Julio said in the webinar, "bad AI doesn't smell" — it can appear correct while being fundamentally wrong. Third, most AI tools for data analysis are built on general-purpose LLMs and not tailored specifically for data analysis and visualization. They often lack the context needed to evaluate particular datasets and edge cases. Finally, there’s an institutional memory gap. Organizations invest significant resources into efforts and decisions whose rationale is quickly forgotten. AI has a real opportunity to dramatically improve how corporate knowledge and principles are communicated and operationalized. ### What do you think the biggest untapped opportunity is as it relates to AI in the data workflow? At Observable, Julio said, we view AI as a human accelerant that reinforces human reasoning and perception for data professionals. Transparency is an area of untapped potential: we need inspectable AI that explains its work, so that users can track and interact with results. Marisa added that while AI may handle many admin tasks, freeing humans from unwanted work, it still requires human discernment and reasoning. She suggested AI could boost data literacy by helping stakeholders collaborate better with data teams. By using AI to explain the data work, the process becomes more transparent, trustworthy, and understandable. Julio pointed out that data literacy is multimodal, requiring stakeholders to understand the data, and data teams to educate collaborators on how to interpret visualizations. If data teams simply assume that charts are commonsensical, they end up undervaluing data work. He sees an opportunity in building a richer dialogue around data work, emphasizing that data professionals must focus on the human elements of learning and exploration: > There is massive underinvestment and underappreciation in the role of advanced data visualization in making your data immediately more intelligible, interpretable, and subject to interrogation. There's a lot there, with respect to data literacy, that can be helped by contemporary tooling and some of the advances that are happening in AI. Julio Avalos, Co-CEO, Observable ## Changing stakeholder expectations ### How have stakeholder expectations around speed to insight changed? Julio highlighted that stakeholders view decision making as an ongoing process, requiring data to be immediately available rather than delivered weeks later as a finished artifact. But data teams and stakeholders remain siloed, which slows iteration. Instead, stakeholders want data and proactive insights in the tools they are already using — such as Slack, Microsoft Teams, etc. — so they don’t have to adopt another tool to analyze data. The future of [self-serve analytics](/content/blog/the-failure-of-self-serve-analytics/index.html) will involve more collaboration, facilitated by better tools and integrations with the data team, or interactions with chat agents. ### Where do stakeholders expect to encounter data? Data visualizations are ubiquitous in BI tools, but the performance and expressiveness of data visualizations in business applications lags behind that of consumer-facing apps, Julio said. Data teams must now build visualizations across a wide range of surfaces, and beyond traditional dashboards. The dashboard-centric paradigm for data analysis and BI is fading. The goal shouldn’t be a particular presentation mode: it should be better human understanding and insights that enable confident decision making. Marisa emphasized the importance of meeting people where they are, and noted that role hybridization is a major shift in the broader workplace: > More people and functions are having to look at and understand the data, have data literacy, and share that information with people across the entire company. That is going to be a big shift as roles become more hybridized. Not only do we have to get data to people — the right data, at the right time — but also more people are being exposed to it and expected to do something with it. Marisa Morby, Director of Research ### How have stakeholder expectations changed around the quality and performance of visualizations? Advanced visualizations are key to unlocking data potential, Julio asserted, because they’re often the best way to find answers in large, complex data. But many incumbent BI tools fail to provide the interactive, expressive charts required to uncover hidden insights. Good visualizations help stakeholders engage with data, mitigating [dashboard rot](/content/blog/five-strategies-fight-dashboard-rot/index.html), which costs organizations millions and destroys confidence in data teams. Focusing on making data more intelligible and engaging through visualization can greatly improve corporate [data culture](/content/blog/how-to-build-data-culture/index.html). ## Conclusion During this [fireside chat](https://www.youtube.com/live/nzpF_j_GAmQ?si=uhe1nZvQbp2sViaO), Julio and Marisa discussed how data teams can navigate the changing BI and analytics landscape by moving away from dashboard-driven workflows and toward continuous, [collaborative analysis](/content/blog/collaborative-analytics/index.html) that meets stakeholders where they are. AI is already reducing mundane tasks, and stakeholders are upping the ante with greater expectations around speed, accessibility, and visualization quality. Data teams can rise to the occasion by combining advanced data visualizations, collaborative analytics, and thoughtful AI integration. For a deeper dive into our research on how data teams are adapting, read our blog post recapping our survey on the [State of BI and Data Analytics](/content/blog/state-of-bi-and-data-analytics-2026/index.html). \]\]>https://observablehq.com/blog/4-ways-to-avoid-common-gotchas-in-data-analysis https://observablehq.com/blog/4-ways-to-avoid-common-gotchas-in-data-analysisThu, 05 Feb 2026 00:00:00 GMTData analysts are under pressure to churn out more insights, faster. But in the rush, they can skip over critical background research, exploration, and scrutiny needed for robust and trustworthy outputs. In this post, we outline practical habits that early career data practitioners should build into their workflows to avoid common gotchas in data analysis. Analysts can deliver more accurate and trustworthy insights with less thrash by taking a beat to deeply understand the data (what’s there, and what’s missing!) before diving in, keeping close eyes on each data wrangling operation, and scrutinizing AI-generated outputs. ## **Know before you code** We know the feeling: getting your hands on a new dataset is exciting, and you’ve got a whole queue of requests to get through. These pressures can tempt analysts to immediately write code and build visualizations without fully understanding the data. If you don’t have a robust understanding of both the data and the questions you need to answer before jumping in, you’re putting yourself on a path to misinformed analyses and wasted effort. Start by familiarizing yourself with the data. Get a broad overview of what’s in your database, perhaps using [AI to help with quick data profiling](/content/blog/three-ways-to-use-ai-for-exploratory-data-analysis/index.html). Then, dig further into the metadata to answer: - What does each row represent? - What does each variable represent? - What is each variable type (categorical, numeric, temporal)? - What are the variable units? - How fresh is the data, and how frequently is it updated? - Are there duplicate records? Depending on your database and analyses, you might also ask questions about the data collection, quality assurance, and more. Just as important is understanding why this analysis matters, and for whom. Who will use the output from this work? _How_ will it be used? Could they already find what they need in an existing report or dashboard? Taking the time up front to answer these questions may feel like it’s holding you back from pressing analyses, but it will save you from costly mistakes and unused work down the line. ## **Don’t miss missing values** When working with data, most analysts understandably focus on the data they have _—_ not on what’s missing. When missing values are ignored, your model outputs may be biased. As a result, your analyses might lead to invalid or overstated conclusions. Investigating missingness should be a non-negotiable part of exploratory data analysis. Build these habits into your data exploration, so you don’t get bit by missing values. ### **Know how missing values show up in your data** Someday, we may live in a world where there are universal standards for recording missing values. But we’re not there yet. Missing values are entered in a number of creative ways: as blank cells, NAs, impossible values for the field (e.g., -999 for a team size), character strings (such as “missing” or “no value”), and more. On top of that, missing values don’t always mean the same thing across datasets, or even across fields. For example, does a missing value mean the value doesn’t exist? Or that the value exists, but couldn’t be recorded? Or the value exists, but is undefined or unknown? To make informed decisions about handling missing values, you need to know _how_ they are represented, and _what_ missingness actually means. Look to the metadata to confirm how missing values are stored, and how they should be interpreted. If missing values aren’t documented, investigate further: ask members of your team who have worked with the data previously, and check for the usual suspects in each field (blank character strings, _null_ s, etc.) to find the answers. Once you do, be sure to update the documentation — your future self, and your colleagues, will appreciate it! ### **Explore the prevalence and patterns of missing values** Next, get a sense of just how muchof your data is missing. Start broadly by finding the proportion of missingness for the entire table. Then, narrow down your exploration to assess missingness by variable, and then by group or category within each field. Doing so can reveal non-random or disproportionate missingness (at times indicating _missing at random_, or _missing not at random_ mechanisms), which, if left ignored, can lead to biased outputs in downstream analyses. Many data visualization tools apply listwise deletion by default, which omits an entire record if any variables used in a chart channel are missing. So, don’t assume you’ll see missing values in a chart. Be sure to check if and how missing values are handled in whichever visualization tool you’re using. ## **Never stop exploring** Even with rigorous data collection standards and quality checks, data wrangling is an unavoidable part of any analysis. Whether you’re cleaning up inconsistent entries, filtering to assess a particular subset, or aggregating hi-frequency or resolution values into larger bins, your data will undergo some transformation between raw table and final report. When multiple data wrangling steps are chained together without checking the intermediate outputs, it’s not always clear if or how something has gone awry. For example, you might botch a unit conversion, which becomes less obvious once you’ve aggregated values by group in the next step. Even minor data entry differences like capitalization (e.g. "enterprise" versus "Enterprise"), whitespace, or punctuation can make it easy to unintentionally filter out entries you meant to include. While these mistakes are often straightforward to fix, they can also be easy to miss if you don’t keep eyes on your data at each step. The solution is to inspect the resulting data after each operation, but doing so can be time consuming. Modern BI tools are evolving to keep your data visible by default, so you can see changes to the data at each step without building new views from scratch. When visual summaries available throughout the data analysis pipeline, analysts can identify mistakes earlier, and collaborators can more confidently interpret, engage with, and trust analyses regardless of their coding experience. ## **Always scrutinize AI** The growing role of AI in data analysis is undeniable. Data practitioners and teams are rapidly embracing AI as a tool to supercharge data profiling, exploration, wrangling, app development, and beyond. But AI-powered data analysis also comes with risks: AI hallucinates, makes mistakes, and lacks context and domain expertise to deliver trustworthy analyses on its own. To mitigate these risks and ensure sound results, data analysts must approach AI with scrutiny. We can't have AI work in a black box — it must operate transparently, allowing for close human scrutiny by those with the necessary skills and understanding to responsibly validate its results. AI can’t replace the need for coding knowledge, contextual understanding, and domain expertise. Keep in mind that just because some AI-generated code _runs_ doesn’t mean that _it’s doing the right thing_. In most cases, analysts should go even further by validating results against the underlying data, and gut-checking the results based on their own judgment and domain knowledge. ## **Learn more** The habits we describe above take time, but they don’t slow analysis down. By doing the necessary research beforehand, and closely inspecting each data wrangling operation and AI-generated output, analysts can avoid costly mistakes that — if eventually discovered 🤞— take much more work to address after-the-fact. Looking for more resources to get started with robust data exploration and analysis? Check out our recent posts: - [Three ways to use AI for exploratory data analysis](/content/blog/three-ways-to-use-ai-for-exploratory-data-analysis/index.html) - [How to explore data: a checklist for navigating datasets](/content/blog/data-exploration-checklist/index.html) \]\]>https://observablehq.com/blog/state-of-bi-and-data-analytics-2026 https://observablehq.com/blog/state-of-bi-and-data-analytics-2026Mon, 26 Jan 2026 00:00:00 GMTBusiness intelligence is going through a period of significant change. Advances in AI, growing demand for better self-serve analytics, and shifting expectations about how data fits into everyday work are reshaping how companies think about and use data. In this post we outline key trends shaping BI and analytics in 2026, from the rise of agentic and AI-powered analytics to the growing importance of trust and collaboration. Over the past year, we spoke with data practitioners and customers about their experiences using current BI tools: where they fall short, why confidence in AI-generated insights remains low, and what data teams can do to close the gap between promise and practice. Together, these trends point to a future where analytics moves beyond static dashboards toward interpretable AI, collaborative workflows, and insights that meet people where they work. ## Agentic and AI-powered analytics AI is fundamentally transforming how businesses collect, analyze, and understand their impact through data. As BI and analytics platforms introduce more advanced agentic analytics capabilities, data teams and stakeholders are starting to move beyond asking AI about what happened in the past. Instead, they are beginning to rely on AI to suggest actions they should take next based on the data. Even so, there’s a gap between the potential value of agentic AI and the real-world constraints that limit AI adoption. To better understand what’s holding AI adoption back, we surveyed data practitioners in late 2025. When asked how confident they were in the accuracy and reliability of AI-generated insights in their current BI tools, only 10% said they were somewhat or very confident. Everyone else reported feeling neutral to very unconfident. This highlights a clear disconnect between the value AI tools promise and the trust users place in their outputs. Their skepticism around AI for data analysis isn’t unwarranted: AI unavoidably hallucinates, misinterprets requests, and makes mistakes. The problem is that when AI works as a black box, users can’t see how an answer was produced, which makes it hard to verify whether a response is correct or appropriate for the situation. That’s why, at Observable, we are taking a [transparent and human-centric approach to AI](/content/blog/playing-safely-with-fire-building-interpretable-ai-for-data-analysis/index.html). Observable AI is designed to “show its work” by producing interpretable and editable queries and responses that keep analysts in the driver’s seat. ## The growing importance of trust Confidence in analysis can quickly erode when there are questions about data quality or how raw data was transformed into a chart, dashboard, or report. Since mistrust is a major barrier to AI adoption for data analysis, improving the accuracy and verifiability of AI’s work will be increasingly important. Verifiability needs to be more thoroughly baked directly into AI tools and workflows. For example, when answering a data question, AI should return a query with clear provenance that a user can inspect. Giving users the ability to audit AI’s work is essential for driving adoption and closing today’s trust gap. Another area where teams can increase trust in output is by making it easier to track data transformations — whether performed manually or by AI-generated queries. [Making data visible in charts and tables](/content/blog/visual-data-analysis/index.html) throughout the analysis process can reveal new questions, anomalies, mistakes, and misunderstandings earlier. It also makes results easier to interpret across a wider audience. Accessible, interactive data exploration fosters trust because users have the agency to dig into the data on their own. When a user can independently click on a bar or brush over a cluster to investigate different slices of the data, ask follow-up questions, or even have AI help explain what they’re seeing in context, they build confidence in their understanding and shared results. There is an important role for interactive data visualization, whether created by AI or a human, in increasing trust and transparency in analyses. ## From self-serve analytics to collaborative analytics Stakeholders increasingly want to explore data on their own. In part, this is because data teams are often overwhelmed by inbound requests, creating slow response times. More importantly, data can’t become part of everyday decision-making if every new question requires a handoff. This shift toward self-serve analytics is a positive one, but it comes with tradeoffs. Many self-serve analytics tools limit deeper exploration or advanced, custom chart creation, resulting in shallow insights. Even worse, self-serve tools can unintentionally reinforce silos by creating the perception that stakeholders don’t need to interface with data teams at all. [Collaborative analytics](/content/blog/collaborative-analytics/index.html), on the other hand, seeks to create more points of connection between data teams and their stakeholders by involving stakeholders throughout a project, instead of just tapping them for a random question or copy edits on a final report. This approach brings valuable perspectives and expertise to the entirety of the data analysis workflow from ideation to final data product. As Observable’s co-founder and CEO Mike Bostock put it, “Isolating roles is counterproductive; we should instead bring people together more efficiently.” With the right tools and processes, data teams that embrace collaborative analytics create guardrails for stakeholders to empower their exploration. ## Breaking charts out of the BI dashboard Charts are already ubiquitous in consumer technology, and their presence in business software will only continue to grow. Embedded analytics is expanding the domain of charts and insights beyond traditional BI dashboards, and into the tools people already use — from their company portal to messaging apps like Teams or Slack. These use cases require responsive, lightweight charts that [work well in small spaces](/content/blog/big-insights-small-spaces/index.html). An accelerator for this shift will be agentic AI. As chatbots and AI assistants become embedded in everyday workflows, they’ll proactively surface relevant charts and insights at the moment of need. Bringing the chart to the user keeps insights visible and timely, prevents dashboard rot, and accelerates data-driven decision making. Most importantly, it allows employees to engage with data without breaking their flow or switching tools. ## What this means for data leaders Underlying each of these trends is an important shift in the role of data. The success of analytics initiatives isn’t determined by how many dashboards are created or how many views those dashboards received. Rather, success is evidenced by the role that data plays within the daily activities and decisions of employees. It is about democratizing data while maintaining consistency and guardrails, giving employees the tools they need to increase their own adoption of data in their respective roles, and embracing tools and practices that build trust in AI- or human-generated insights. \]\]>https://observablehq.com/blog/draft-and-customize-queries-even-faster-with-streamlined-text-to-sql https://observablehq.com/blog/draft-and-customize-queries-even-faster-with-streamlined-text-to-sqlThu, 22 Jan 2026 00:00:00 GMTIn December we announced big [updates to the SQL node](/content/blog/recent-updates-to-the-sql-node-in-observable-canvases/index.html) in [Observable Canvases](/content/platform/canvases/index.html) — our new collaborative whiteboard for flexible data exploration and visualization on an infinite canvas — that improve the SQL writing, editing, and troubleshooting experience. Now, we’re excited to share how we’ve streamlined AI in the SQL node, making it even faster to draft and customize queries with text-to-SQL prompts. Read on to learn about our new AI integration within SQL nodes, directly editable SQL queries, and visual diffs that leave no doubt about what’s changing as you iterate. ## **A new AI toolbar right in the SQL node** Previously, users could write prompts in the AI toolbar at the bottom edge of their browser window to generate a new SQL query. This wasn’t ideal: writing AI prompts outside of the SQL node created too much distance between the query and AI, leading to a fragmented experience spread across different places on the canvas. Now, we’ve added an AI toolbar directly into the SQL node, which lives immediately below the query. This eliminates the physical and mental gap between your AI requests and what’s generated by it, so you can more easily evaluate and update the output. Open the AI toolbar in a SQL node by clicking the icon in the top left of the side panel, or using the Cmd-k shortcut (Ctrl-k for Windows). Write and send your request, inspect the code, then run the query and see the resulting data — all in one place on the canvas, as shown below. AI is now integrated directly in the SQL node so you can write, see, execute, and inspect the output of AI-generated SQL in the same place. ## **Use AI to update existing SQL queries** Another limitation of our earlier AI for SQL experience in canvases was that AI could only create _new_ queries in _new_ SQL nodes — it could not edit existing queries. This resulted in unused and redundant SQL nodes cluttering the canvas. Plus, it hindered AI-assisted iteration because prompts always created a brand new query instead of making tweaks to an existing one. With our recent updates, you can now use AI to edit existing SQL queries. Below, a user makes two updates to an existing query, first changing to revenue aggregation from monthly to yearly totals, then excluding the Electronics & Media category: As you edit, you might also want to take a look back at previous requests. Hover over the truncated prompt text in the AI toolbar to explore your prompt history: ## **AI suggestions highlighted by visual diffs** AI can be a powerful tool to accelerate data wrangling and analysis, but only if users have full confidence in the results. In canvases, we’re taking a [human-centric approach to AI](/content/blog/playing-safely-with-fire-building-interpretable-ai-for-data-analysis/index.html) that produces verifiable, interpretable outputs allowing for scrutiny and correction. In line with that philosophy, we want to ensure that when AI makes changes to existing SQL code, the changes are easy to find, inspect, and update. With our new visual diffs, there’s never a question about what will be added to, or removed from, your SQL query. AI-suggested SQL updates are shown in green (for added code) or red (removed code), as shown below. A user can choose to accept or reject the AI suggestions before re-running the query. ## **Advancing AI-powered data analysis in Observable Canvases** Here, we’ve highlighted several recent updates that improve how analysts can use AI to supercharge their data wrangling and analysis in SQL nodes in [Observable Canvases](/content/platform/canvases/index.html). With a new AI toolbar that operates directly in the SQL node, the ability to edit existing queries, and visual diffs that leave no doubt about how AI suggestions will change your code, we now offer an even more streamlined and trustworthy text-to-SQL experience. Want to follow along as we built out the premiere platform for AI-powered data exploration, visualization, and analysis? Bookmark or subscribe to our [release notes](/content/release-notes/index.html), and [sign up for our monthly newsletter](/content/?modal=newsletter/index.html) to stay up to date with what’s new in canvases. \]\]>https://observablehq.com/blog/democratizing-advanced-interactive-data-visualization-in-observable-canvases https://observablehq.com/blog/democratizing-advanced-interactive-data-visualization-in-observable-canvasesFri, 16 Jan 2026 00:00:00 GMTThe qualities that make for the most engaging and valuable data visualizations are often the hardest to implement from scratch. Features like interactive filtering, responsiveness to adjust for different screen sizes, extreme value handling, and optimized performance can have a huge impact on how stakeholders experience your charts and dashboards. At the same time, adding these advanced chart features can be a big lift, making the last 5% of dashboard development take up 95% of your effort. Unfortunately, this often means that exceptional visualizations and dashboards are only possible for advanced developers with the skills (and time!) to code them up. Everyone else is left filing requests with their organization’s data visualization developer (if that position exists), or settling for shallow, underwhelming dashboards that end up [rotting away](/content/blog/five-strategies-fight-dashboard-rot/index.html). We want to change that. In [Observable Canvases](/content/platform/canvases/index.html) — our new collaborative whiteboards for data exploration, analysis, and visualization — we’re democratizing interactive, advanced charts. Read on to learn about some of the expert-level chart features that users get for free in canvases. ## **An expanding chart library from expert developers** Our growing chart library already includes state-of-the-art advanced charts like Sankey diagrams, treemaps, beeswarm charts, streamgraphs, and grid cartograms — all possible to build in just a few clicks. All charts in the library are developed and tested by Observable data visualization engineers. With meticulously designed out-of-the-box charts, you avoid time spent tweaking chart details to get your visuals presentation-ready. For each chart, UI options let you quickly define color and size channels, update time series intervals (e.g., daily, weekly, or monthly aggregation), switch between interpolation algorithms, and more. Selected charts in the Observable Canvases chart library. ## **Built-in visual filtering** Visual filtering allows a viewer to explore different subsets of data across views by interacting directly with a chart or table (as opposed to creating new subsets with code). When a group or segment of data is selected in an upstream chart — usually by brushing over continuous values, or selecting discrete categories — downstream charts automatically update to reflect the same subset. This makes visual filtering a useful, lightweight way to test and explore relationships for different slices of the data. Unfortunately, it is also notoriously challenging to implement from scratch. In canvases, visual filtering is available in table nodes and in many of our large charts including density charts, scatterplots, bar charts, choropleths, and grid cartograms. When you brush or select a subset of data in an upstream node, downstream charts are automatically updated to reflect the added filter. For example, when a user brushes over the density chart below (left), the downstream bar chart is automatically filtered to the selected subset of data records: Similarly, you can select or deselect multiple categories to filter by discrete groups using the following shortcuts (swap Ctrl for Cmd in Windows): - **Cmd-click:** select individual categories - **Option-click:** deselect individual categories - **Click-shift-hold:** select contiguous categories - **Option-click-shift-hold:** deselect contiguous categories In the horizontal bar chart below, a user selects (purple) or excludes (orange) a set of product categories to limit records included in the downstream beeswarm chart: ## **Responsiveness makes charts a good fit for any screen** Have you ever made an exquisitely formatted chart, only to find it’s unreadably small or horribly distorted when viewed by collaborators on another device? Yeah, us too — and it’s a hassle to fix after-the-fact. All charts in the canvas library are responsive by default. As the chart area changes, charts adjust automatically to maintain readability and polished styling on any device, as shown for the activity timeline below: Some charts even have responsive resolution, meaning the level of aggregation changes to show more detail when space allows. For example, the histogram bin width adjusts automatically as the chart width updates: Notice how the bin width automatically adjusts as the histogram width changes, delivering greater resolution as more space becomes available. ## **Extreme value and “others” handling** There’s no single right way to handle extreme values and small categories in data visualization. In some cases, those may be the _most_ important data points to dig into. More often, the effect of including “out there” observations is to extend chart axes far beyond the bulk of your data, which then gets compressed into a small corner of your chart. Options like log transformation can help, but can also make interpretation challenging for many viewers. In canvases, we’ve built smart outlier and “others” category handling into large charts. Our strategy helps you focus on a clear view of the _vast majority_ of your data, while keeping extreme values or unseen categories visible so you can dig in further when you need to. Examples of our implementation are shown below for a horizontal bar chart and [histogram](/content/blog/designing-better-histograms/index.html) are shown below: ## **Performant charts, no matter the size of your data** Companies are collecting increasingly large volumes of data that, in the right analysts’ hands, has the potential to unlock valuable business insights. But trying to visualize millions of records (or sometimes, far fewer) can slow charts and dashboards to a crawl in traditional BI tools, making them all but unusable. At the same time, plotting millions of individual data points is rarely useful: too many marks leads to noisy, overwhelming visuals that make it hard to focus on the most important signals. To keep charts fast and clear in Observable Canvases — even when you’re working with huge datasets — we run optimized queries that return a smaller, pre-aggregated version of your data that is still at high-enough resolution to produce rich charts. Learn why charts built on big datasets tend to be slow, and how our approach delivers rich _and_ performant charts, in our [recent blog post](/content/blog/big-data-fast-data-visualizations/index.html). ## **Follow along as we expand the premier chart library for business intelligence** Data analysts shouldn’t be limited to static, shallow visualizations and basic chart types. In canvases, we’re making advanced features like interactive visual filtering, responsiveness, extreme value handling, and performance available for everyone — not just seasoned developers. Learn more about how Observable Canvases can help your team explore and visualize data [here](/content/platform/canvases/index.html). To stay up to date as we add new charts and features, [sign up for our monthly newsletter](/content/?modal=newsletter/index.html) and bookmark or subscribe to our [release notes](/content/release-notes/index.html)! \]\]>https://observablehq.com/blog/observable-2025-year-in-review https://observablehq.com/blog/observable-2025-year-in-reviewWed, 17 Dec 2025 00:00:00 GMT2025 was an exciting and transformative year for Observable. As we head into the new year, here’s a look back at the products we launched, the events we hosted, the community creations that inspired us, and what we’re excited for in 2026. ## **What we launched** ### **Observable Canvases** After a busy spring spent building behind the scenes, we were thrilled to announce [early access to Observable Canvases](/content/blog/introducing-canvases-early-access/index.html) in mid-April. Observable Canvases are browser-based collaborative whiteboards for [data exploration](/content/blog/eight-features-supporting-fluid-data-exploration-in-observable-canvases/index.html) and analysis. In canvases, analysts can work flexibly between [SQL](/content/blog/recent-updates-to-the-sql-node-in-observable-canvases/index.html), UI, and [AI](/content/platform/ai/index.html) to transform and analyze data across the infinite canvas, then select from our state-of-the-art chart library to get from raw data to [fast, interactive visualizations](/content/blog/big-data-fast-data-visualizations/index.html) in minutes. Learn more about [Observable Canvases](/content/platform/canvases/index.html). Since our April launch, we’ve been adding [more advanced charts](/content/blog/new-advanced-charts-in-observable-canvases/index.html) to the library, improving performance, developing a new approach to [dashboarding](/content/blog/dashboards-in-observable-canvases/index.html) right from a canvas, and acting on feedback from early testers. Expect big news and exciting releases around Observable Canvases in 2026! ### **Verifiable AI** AI can take some of the tedious grunt work out of data exploration, offering a faster way to [profile and wrangle data, prototype charts, and kickstart new analyses](/content/blog/three-ways-to-use-ai-for-exploratory-data-analysis/index.html). But the risks of AI in data analysis are real: in many tools, AI works in a black box that makes it difficult for users to realize when AI makes an error or does something unexpected with your data. In July, we announced integrated, transparent, and verifiable AI that speeds up your data work in Observable Canvases, while keeping you in the driver’s seat. Read Mike Bostock’s [AI launch post](/content/blog/playing-safely-with-fire-building-interpretable-ai-for-data-analysis/index.html) to learn about our human-centric AI principles and how we’re thoughtfully building AI for data analysis. ### **Notebooks 2.0 and Notebook Kit** The long-awaited next generation of Observable Notebooks — local, file-based, and supporting vanilla JavaScript — landed this summer! We simultaneously released: - **Notebook Kit:** An open file format for notebooks, and accompanying open-source tooling for generating static sites. - **Notebook Desktop:** A macOS desktop application for editing notebooks as local files. Build locally, code in vanilla JS, and commit files to git with Notebooks 2.0. Haven’t tried Notebooks 2.0? Check out the [documentation](/content/notebook-kit/index.html) to learn more and get started! ## **Events we led** We hosted a number of events this year, from product-focused webinars led by the Observable team, to inspiring panel discussions featuring top data visualization developers and data journalists. Did you miss any of our events as 2025 went flying by? No problem! You can watch the recordings any time: - 📣 [Observable Canvases, a first look](https://youtu.be/0EaRdtm2X8A): Shortly after our April announcement, we hosted our first public demo of Observable Canvases! Canvases have evolved substantially since, but the core workflow for fluid, branching data exploration and fast interactive charts remains. - ✨ [What’s new in Observable Canvases](https://www.youtube.com/watch?v=Gfp5CssnKlU): Product experts and engineers from the Observable Team shared new canvas features like advanced charts, extended sharing options, more database connectors, and improved workspace management settings. - 🎨 [Art & Science: Blending analysis and creativity in data visualization](https://www.youtube.com/watch?v=ciV3q3wJEVs): An inspiring and personal discussion about balancing creativity and analytical rigor in data visualization, featuring Kelsey Nanan, Tanya Shapiro, and Shirley Wu. - 📰 [Storytelling with graphics — from raw data to reader impact](https://www.youtube.com/watch?v=_dh4gRkQiRw): A chat with data journalists Kavya Beheraj (Axios), Ben Welsh (Reuters), and Jared Whalen (The Pudding) on crafting stories from messy data, asking the right questions, and the need for empathy and creativity in news graphics. ## **Community creations we loved** This year, as always, we were inspired by the creativity and ingenuity of the Observable Community. From novel notebooks to impactful data apps, we loved seeing what you all made with Observable. A look at the top notebooks from 2025 reveals a mix of creative flare, interactive educational resources, and beautiful maps. Here are a few of the most popular notebooks shared this year: - 🧑‍🎨 [Hello custom marks with Plot](/content/@jwolondon/hello-custom-marks-with-plot/index.html) by [Jo Wood](/content/user/@jwolondon/index.html): Follow along with this exceptional walk-through to create custom marks for use in Observable Plot charts. Jo also shared a popular [Making Plot Sketchy](/content/@jwolondon/making-plot-sketchy/index.html) notebook that adds a hand-sketched look to Observable Plot charts! - 🗺️ [Bivariate maps with Plot](/content/@joewdavies/bivariate-map-with-plot/index.html) by [Joe Davies](/content/user/@joewdavies/index.html): Explore the relationship between temperature and precipitation for different geographic extents in these eye-catching maps, made with Observable Plot. - 🍿 [Kernel Density Estimation (KDE)](/content/d/cf8265f4c08dd426/index.html) by [Pierre Le Bras](/content/user/@plebras/index.html): Stats teachers, take note! This educational, interactive notebook introduces KDE, and lets viewers explore the impact of different Kernel functions, bandwidths, and more. - 🏢 [Norwood, Bronx – VTX](/content/d/080e3912cbec34d1/index.html) by [Vivi Li](/content/user/@vivi-li/index.html), Tristan Guan, and Xenia Villaver: A stunning, interactive city map to explore “the potential for vertical extensions in Norwood in the Bronx as a strategy to help alleviate the need for housing in New York City.” We also highlighted the work of five Observable creators on our blog. Check out our Creator Features to learn how they’re building charts and data apps that help users answer hard questions, discover new ones, and engage with complex data: - 🏘️ [From Urban Science to data storytelling: Inside the work of Kelsey Nanan](/content/blog/kelsey-nanan/index.html) - 🌍 [We can always talk about maps](/content/blog/maps-and-data-visualization-with-fil-riviere/index.html) with Observable Data Visualization Engineer Fil Riviére - 🌱 [Mapping the future of agriculture with AgroPatterns](/content/blog/agropatterns-observable-greenhouse-analytics/index.html) with Gabriel Koch - 🔎 [Making science visible: Dr. Philip Bogden on bringing data visualization to cancer research](/content/blog/making-science-visible-dr-philip-bogden-bringing-data-visualization-to-cancer-research/index.html) - 📈 [The anatomy of an impactful data visualization with Karim Douieb](/content/blog/anatomy-of-an-impactful-data-visualization-with-karim-douieb/index.html) ## **What we're excited for in 2026** Looking ahead, we have big plans for the new year. Here are some things we’re excited about, straight from the mouths of Observable team members: > We've got an exciting lineup of webinars and feature announcements taking shape in early 2026, so stay tuned for event updates! And, we'll be shipping a wave of new educational resources to help folks get up and running in Observable Canvases. We're as inspired as ever by the Observable Community, so you can look forward to more community highlights and Creator Features in the new year. Hanna Woodburn VP of Marketing * * * > We're working on exciting advances in Observable Canvas’ charting capabilities, bringing in some of your favorite charts from the D3 and Plot galleries, plus adding more options, controls, and details to our existing charts. We've overhauled our process to speed up chart development, making it easier to build and test new charts so we can deliver more than ever. 2025 was just the beginning of on-demand charts in canvases — 2026 is about taking it to the next level! Paul Buffa Director of Product * * * > I’m looking forward to spending more time learning from our users and understanding their core problems and where our product can better support them. We want to incorporate those insights into design improvements that strengthen the overall user experience. I'm looking forward to continuing to build a product that addresses core user needs and makes people excited to explore and share data insights with their teams. Marisa Morby Director of Research * * * > We’ll continue to improve performance and stability so that users get a consistent, delightful experience across canvases, notebooks, charts, and dashboards. I'm looking forward to having a solid foundation for the chart library going into 2026, so that we can expand the charts we offer and maintain the same polished experience across Observable tools. Seth Engelhard VP of Engineering Want to keep up with what’s happening at Observable? Sign up to receive our [monthly newsletter](/content/?modal=newsletter/index.html). We’ll see you in the new year! \]\]>https://observablehq.com/blog/webinar-recap-storytelling-with-graphics-from-raw-data-to-reader-impact https://observablehq.com/blog/webinar-recap-storytelling-with-graphics-from-raw-data-to-reader-impactThu, 11 Dec 2025 12:00:00 GMTNewsrooms have long been at the forefront of data visualization, turning messy datasets into clear, powerful stories that resonate with broad audiences. On December 4th, leading data journalists, [Jared Whalen](https://www.linkedin.com/in/whalenjared/) (Polygraph/The Pudding), [Ben Welsh](https://www.linkedin.com/in/palewire/) (Reuters), and [Kavya Beheraj](https://www.linkedin.com/in/kavyabeheraj/) (Axios), gave us a peek behind the curtain on their process. The hour-long discussion covered topics like knowing if there’s a story worth telling in the data, collaborating with others, and tips for communicating with clarity, urgency, and impact for the broader data visualization community. Here’s our recap of our webinar, “Storytelling with data: From raw data to reader impact.” ## Showcasing real-life stories that made an impact We kicked off the conversation with a “show-and-tell” with each panelist sharing a data visualization they worked on that they felt was impactful. As they walked us through the background and process, they also reflected on key learnings. [Skip to 3:29 to listen to this segment.](https://www.youtube.com/live/_dh4gRkQiRw?si=sezunzZ3ZwTIDQD7&t=209) ### Ben Welsh: Visualizing Los Angeles Fire Department response times Ben shared a map of Los Angeles Fire Department (LAFD) response times he created for The Los Angeles Times after a mayoral candidate in 2012 publicly questioned the speed of 911 responses. To fact-check the candidate’s claims, the team conducted their own analysis with millions of 911 call records. They then used national response-time standards as an external benchmark to objectively evaluate performance. > “In any data visualization, if you wanna get more investigative, \[it’s important to\] have that outside standard or something to measure up against that isn't just like ‘according to you’ or according to its standard deviation in the distribution, but some real benchmark.” Ben Welsh, News Applications Editor, Reuters The map surfaced two major issues: extremely slow response times in wealthy hillside neighborhoods and widespread delays near the city’s borders where fire stations were less accessible. Further analysis uncovered that the longest delays weren’t due to travel time but due to the early call-taking and triage process. The data visualization and accompanying reporting highlighted new findings and sparked significant reforms, such as revamped call-taking procedures and the creation of an internal statistical analysis unit at the LAFD. ### Jared Whalen: Examining reporting on trans issues Jared shared a recent Polygraph project built in partnership with the Trans Journalists Association to “audit” how the news industry covers issues related to transpeople, such as healthcare and state legislation. Using the [Media Cloud API](https://www.mediacloud.org/), the team assembled hundreds of thousands of relevant articles over the past five years. Using a large language model (LLM) classifier, they then sorted coverage into themes. For instance, the dataset surfaced roughly 18,000 healthcare-related articles, which the LLM further clustered into about 150 discrete news events. This helped reveal which stories dominated coverage. The goal, Jared explained, was accountability: giving journalists and readers a way to understand how newsrooms talk about trans-related topics. The resulting interactive let users drill into individual articles, compare coverage by publication and political leaning, and see how attention surges over time through [beeswarm visualizations](/content/@d3/beeswarm/2/index.html) and [time-series views](/content/blog/analyzing-time-series-data/index.html). One of the most crucial parts of the project, according to Jared, involved weaving in community feedback on how coverage should be framed. Because it’s built to run continuously, the dataset updates each month while the team periodically audits and retrains the model. ## Kavya Beheraj: Localizing wildfire-smoke data Kavya walked through a wildfire-smoke visualization she built after the East Coast’s unprecedented orange-sky summer in 2023. The goal was to explain what made that season’s smoke so extreme and what it signals about our “new normal” in air-quality exposure. A key feature of the piece was a localized dashboard that let readers search their metro area and compare recent pollution levels against a decade of historical data. Because Axios has more than 30 local bureaus, the team wanted a way to turn a massive national story into something personally relevant for readers in each city. The visual design showed the most recent readings, how they compared to long-term averages, and whether conditions were considered healthy under federal standards. The team also included practical guidance on protecting yourself from wildfire smoke. By investing in cleaning and preparing the data for local markets, Axios was able to repurpose the information and extend the story’s impact far beyond its initial publication. ## Finding the story in messy, imperfect data We asked our panelists: “When you're handling a messy dataset, what is your first step in figuring out if there is a story worth telling?” [Skip to 25:14 to listen to this segment](https://www.youtube.com/live/_dh4gRkQiRw?si=MI3wdoraVo9vJLaW&t=1514). Jared’s take was to ask why the dataset exists in the first place and understand how it’s used. He continued on to add that referring to documentation or talking with the data manager to get that context should be the first step when you encounter a new dataset. > “It's very tempting to dive right into it and inject your interpretation of what it could tell you…But what can often end up happening is you spend a lot of time getting the data to what you think it should look like, only to find it doesn't actually tell you what you think.” Jared Whalen, Senior Journalist/Engineer, Polygraph/The Pudding Ben cautioned that even a small misunderstanding of a row or column can skew your entire analysis. He advised our audience to go back to the fundamentals of the table itself, slowing down to ask what a row even represents. > “What is a row? And I mean that in the most philosophical and Socratic sense. What does each row represent? What is it recording? What does each column indicate?” Ben Welsh, News Applications Editor, Reuters Kavya followed up to reinforce the importance of understanding the data at basic level and adding, “Trust your gut. If something is confusing to you, then it's probably going to be confusing to other people.” ## A peek under the hood: Process and workflows When it comes to how visual stories actually get made, the panelists agreed on one thing: there is no single newsroom workflow. Each team blends tools, roles, and processes differently, depending on the story, the deadline, and the people involved. [Skip to 46:55 to listen to this segment](https://www.youtube.com/live/_dh4gRkQiRw?si=bMv-2CoyTvAFXn-y&t=2815). Ben noted that newsrooms rarely operate like traditional product teams with a clear division of labor. Instead, they rely on a flexible mix of practices and tooling. Most custom visual stories are ultimately built with some combination of static-site tooling and templating systems (“Frankenstein” stacks, as he put it). Jared emphasized that deciding whether to design in Figma or directly in code is always case-by-case. Data-heavy stories often need to be shaped by the data itself, making early mockups risky. In other cases, having clean design prototypes upfront can dramatically speed up development. Bespoke visuals rarely follow a predictable formula. Kavya added that even within a single newsroom, workflows can vary by person. One teammate might explore data in R, and another in a spreadsheet, but they “all end up in the same place.” She continued to describe that a lot of the processes are “cobbled together” and reflect what they personally think is the best approach at the moment. ## Cross-disciplinary lessons from data journalism We tackled how principles of data journalism can apply to business and product teams. Kavya recalled previous experiences building dashboards that she’s “not sure anyone actually used or saw.” She highlighted a key challenge — the assumption that all of the data points matter — and encouraged our audience to instead take a step back and consider the users of the dashboard, how it will be used, and how it will be maintained. [Skip to 30:07 to listen to this segment](https://www.youtube.com/live/_dh4gRkQiRw?si=7MYTX74hPlb5cLmt&t=1805). When the subject collaboration was introduced, the panelists shared two insightful takeaways: let the story be the north star and remember to let others contribute and shine. > “It helps to let the story steer the ship…Every person who's contributing to this story is supporting the whole. The visual has to work with the words, it has to work with the style, and it has to work with the social strategy. Our mission is to produce the best story possible and to tell a specific narrative. It is humbling and also helpful to have that to fall back on…” Kavya Beheraj, Associate Editor of Data Visualization, Axios ## How technology and AI are reshaping data storytelling The panelists agreed that the landscape for data visualization tooling has never been more accessible. Over the past decade, the rise of open-source tools, cloud-based workflows, and even data visualization communities has dramatically lowered the barrier to building bespoke visual stories. Tools like Observable make it easier to explore data, build prototypes, and ship polished visuals. As is expected, AI is beginning to reshape the data workflow. Ben shared how Reuters now uses an internal chatbot capable of generating Datawrapper charts on command, while Kavya described Axios’ experiments with AI-assisted data triage to help reporters assess whether a dataset is clean or meaningful enough to visualize. Across the board, the panel emphasized that AI excels at the “monkey tasks,” while human judgment still governs narrative decisions. ## Closing thoughts At the end of the hour, our audience got an insightful peek behind the scenes at how data journalists work: grounding stories in rigorous analysis, localizing national issues to make them meaningful, and collaborating with others. They shared how technology and AI are reshaping the craft and how each team adapts their workflows to the needs of the project. Want to hear more from our guests? Watch the [full webinar recording here](https://youtube.com/live/_dh4gRkQiRw?feature=share). Explore additional work by these panelists at their website or on Observable: Jared Whalen ( [website](https://www.jaredwhalen.com/), [Observable](/content/@palewire/index.html)), Ben Welsh ( [website](https://palewi.re/who-is-ben-welsh/), [Observable](/content/@palewire/index.html)), and Kavya Beheraj ( [website](https://www.kavyabeheraj.com/), [Observable](/content/@koffeeya/index.html)). \]\]>https://observablehq.com/blog/recent-updates-to-the-sql-node-in-observable-canvases https://observablehq.com/blog/recent-updates-to-the-sql-node-in-observable-canvasesTue, 02 Dec 2025 00:00:00 GMT[Observable Canvases](/content/platform/canvases/index.html) are modern collaborative whiteboards for fluid data exploration and rich, interactive visualization. In canvases, we’ve implemented UI options for common data wrangling steps like joining tables, filtering rows, selecting columns, and deriving new values. But SQL is still the _lingua franca_ of interacting with databases, and data analysts may need (or simply prefer) to write custom SQL instead of using our UI tools. For that reason, we knew from the jump that canvases needed a SQL node where users could write and edit their own custom queries. While early canvas releases made it _possible_ to write and execute SQL, it was far from an ideal user experience. Over the past several months, we’ve made big improvements to the SQL node. Read on for a quick update on recent changes like syntax-aware autocompletion, better error handling, and UI adjustments that are improving the SQL experience in canvases, plus a preview of what’s coming next. ## **Dialect-aware autocomplete** Previously, users had to look outside of the SQL node to find functions, table and column names, and the correct syntax to use in their queries. From our own experience, we know that usually means too much time is spent troubleshooting, searching through documentation, and asking LLMs for suggestions in a separate chat. Our new SQL node autocompletion avoids those time-consuming side quests, helping you quickly build queries by suggesting table and column names, statements, and functions right in the SQL node as you edit: Different databases can use slightly different SQL flavors. In canvases, autocomplete is dialect aware, helping you to build valid SQL queries for your database type. ## **Error handling** We also recently overhauled error handling in the SQL node for quicker troubleshooting. In earlier canvas releases, you could see that a query error had occurred, but there was no additional information easily in view to help you figure out what went awry — or where. Now, error messages appear just to the right of your SQL code, and provide an interpretable description of what’s breaking while pointing to the exact line number where things went wrong. ## **UI updates** When you’re writing code, even seemingly small UI friction points can add up to a cumbersome experience. Over the past few months we’ve made a number of changes that smooth out previously rough edges in the SQL node, including several commonly requested by canvas pilot testers. These include: - **Variable editor width:** You can now manually adjust the column width of the SQL editor, which is particularly helpful when you want to keep long statements on the same line. - **Fewer clicks to get started:** Previously, it took multiple clicks to start writing SQL in the SQL node. Now, with your node in Edit mode, just click once anywhere in the editing pane and start typing. - **Status highlighting:** You may have noticed the different node border colors in the visuals above. We recently added these visual cues for node status. With a node in Edit mode, the blue, yellow, and red outlines let you know if queries are successful, missing an input, or broken (respectively): ## **Coming soon: AI in SQL nodes** **Update:** The streamlined approach to AI in the SQL node described below is now live in Observable Canvases! Check out [this post](/content/blog/draft-and-customize-queries-even-faster-with-streamlined-text-to-sql/index.html) to learn how tight AI integration, AI-editable queries, and visual diffs make text-to-SQL even faster and easier to verify. Updates to AI in SQL nodes are well underway, and will be available in canvases in early 2026. Currently, Observable AI generates new queries in new SQL nodes, but it won't edit existing queries. This was by design: one of the original tenets of our AI was that it should not edit existing canvas content to avoid risk of AI making unwanted changes to your existing work. However, this approach has its tradeoffs. Adding a new node for each small change can quickly result in a large number of nodes cluttering your canvas, and bogging down performance, with unnecessary queries. Observable Software Engineer Visnu Pitiyanuvath shares how we’re working to solve this issue: > What we're working on now is getting our AI to target a single SQL node. So, if you're working in a SQL node, you can ask questions or request changes just to that node. That way you don’t have to worry about anything else on your canvas being affected. We also want to make it easy to see what AI has changed in your SQL node, so we’re thinking of ways to show the difference. Visnu Pitiyanuvath ## **Learn more** With Observable Canvases we’re making advanced, interactive data visualization possible for everyone — not just experienced developers. We also want canvases to be a hub for visual data analysis that brings analysts, data teams, and stakeholders together in one place. Flexibility is a key part of that vision, which is why we’re improving both the SQL node and UI data wrangling options in parallel. While you _could_ write SQL in canvases even in our earliest releases, recent updates like dialect-aware  autocompletion, clear error messaging, and small but impactful UI tweaks greatly improve the experience. As we keep building out canvases, check in on our [release notes](/content/release-notes/index.html) to stay up to date with what’s new. To learn more about canvases and to request a demo, [click here](/content/platform/canvases/index.html). \]\]>https://observablehq.com/blog/webinar-recap-art-science-blending-creativity-and-analysis-in-data-visualization https://observablehq.com/blog/webinar-recap-art-science-blending-creativity-and-analysis-in-data-visualizationMon, 24 Nov 2025 00:00:00 GMTData visualization sits at a crossroads: part analytical discipline, part creative craft. On Nov 11, we hosted Art + science: Blending creativity and analysis in data visualization, a webinar on how three data visualization leaders — [Kelsey Nanan](https://kan317.wixsite.com/website), [Tanya Shapiro](https://indievisual.tech/), and [Shirley Wu](https://shirleywu.studio/) — strike that balance in their work. What transpired was an engaging discussion covering fundamentals, such as chart choices and collaboration, and the more abstract, such as how personal backgrounds influence data visualization work, creative risk-taking, and even the future of AI in data visualization. Here’s what stood out from the discussion. ## Starting a project: design or analysis first? When asked whether they approach new visualization projects as designers or analysts first, all three panelists gave the quintessential data practitioner answer: “It depends.” [Skip to 4:23 to listen to this segment](https://www.youtube.com/live/ciV3q3wJEVs?si=MJqO2i3q7dIgN4a5&t=263). For Kelsey, personal projects start through a design lens, but on client projects she leans more analytical, focusing on technical constraints and feasibility from the get-go. Tanya, who often wears both hats at different points in a data visualization project, thinks of herself as an analyst while uncovering the message behind the data, then moves into design mode to translate it into the right visual for her audience. Shirley lightheartedly shared a key lesson with our attendees: “Do not start with code first.” Now, she begins with light [data exploration](/content/learn/what-is-data-exploration/index.html), and then fluidly moves back and forth between analysis and design throughout the project. > What I learned from my projects is I can come up with the funnest design, but when I plug the data in, the shape of the data just doesn't fit very well. Shirley Wu on beginning projects with light analysis before design ## Choosing the “right” chart A key part of a data visualization designer’s job is [deciding on the “best” type of chart](/content/blog/right-chart-for-data/index.html) to accurately and effectively represent the data. When asked how they make these types of decisions, our guests agreed that the audience’s data literacy is a major influencing factor. [Skip to 9:35 to listen to this segment](https://www.youtube.com/live/ciV3q3wJEVs?si=_QjEAGr9g-xjaarU&t=575). > I've got to be cognizant of the data literacy level of my audience and meet them with the right graphic vernacular to convey that story. A box plot might be a ‘standard’ chart type for me, but someone working in insurance might not understand it right off the bat. I have to meet them where they're comfortable. Tanya Shapiro on designing with data literacy of her audience in mind Experimentation, speed, and iteration also emerged as themes. Kelsey referenced her use of low-code tools like [RAWGraphs](https://www.rawgraphs.io/) and [Flourish](https://flourish.studio/) to quickly see what the data might look like as different chart types, while Shirley mentioned [Observable Plot](/content/plot/index.html) and even hand sketching as part of her approach. ## Working collaboratively with others When asked about working with clients, all panelists agreed that bringing more voices into a data visualization project can yield better results in the end — but isn’t without its challenges. They each shared practical advice for [working collaboratively](/content/blog/yes-and-collaboration-for-software-developers/index.html): iterate and get drafts out quickly, don’t be precious about your ideas, be thoughtful in choosing who you work with, and always have mutual respect and trust for one another. [Skip to 35:57 to listen to this segment.](https://www.youtube.com/live/ciV3q3wJEVs?si=HzHVrzzMqLn7isqI&t=2157) > Get things out quick and iterate. Don't try to aim for perfection. Oftentimes, I feel like most people know what they don't want versus what they do want. When you put things out in front of them, you're creating conversations. Tanya Shapiro on her approach to working with others ## Personal background as creative inspiration Our panelists all spoke to the role that their individual backgrounds shape their work and each shared a data visualization that is representative of their evolving data visualization journeys. [Skip to 16:29 to listen to this segment](https://www.youtube.com/live/ciV3q3wJEVs?si=gezcUW3kiezFSoQZ&t=989). Kelsey shared a playful holiday card mimicking the ups and downs of an emotional year marked by a torn ACL and months of surgery. She created a ski-slope-themed chart that, according to her, “breaks all the rules,” intentionally incorporating “chart junk” like vertical text and even an animated stick figure collapsing at the bottom. > Data viz was such a great way to memorialize 2024, and I think breaking the rules just helped me get across what a meaningful journey this was. Kelsey Nanan on using rule-breaking design to capture emotional arcs Shirley then shared two deeply personal data “physicalization” pieces: a [dripping-ink performance of her monthly cycle](https://nightingaledvs.com/senses-and-sentiment/) and a piece representing home, belonging, and immigration using tree rings and dissolving paper. Tanya explained that at a young age, she found a creative outlet in music. Learning guitar by studying song covers, then remixing ideas into her own, shaped how she now approaches data visualization: through experimentation and borrowing inspiration from others. Her [NBA beeswarm chart](/content/@tashapiro/nba-beeswarm/index.html) was the outcome of learning [D3](https://d3js.org/) and exploring examples from the Observable [community](/content/trending/index.html). ## Perspectives on AI Our panelists all use AI in their work — with guardrails. They emphasized its ability to generate beautiful charts, but recognized its shortcomings in comprehension, critical reasoning, context, and lived experience. Tanya suggested using AI “as a stencil,” while Shirley spoke about the exciting potential of using AI in analyzing qualitative data. [Skip to 48:23 to listen to this segment.](https://www.youtube.com/live/ciV3q3wJEVs?si=_wubOQkR2LdxYVIy&t=2903) ## Lightning round: advice for early-career practitioners We closed the webinar by asking our panelists to share one piece of advice they wish they had known early in their data viz careers. For Tanya, it was about finding community to accelerate growth. Kelsey spoke on using low-code tools to speed early exploration, and Shirley cautioned against trying to do everything alone. Want to hear more from our guests? [Watch the full webinar recording](https://www.youtube.com/watch?v=ciV3q3wJEVs) and explore their work on Observable: - [Kelsey Nanan](/content/blog/kelsey-nanan/index.html) - [Tanya Shapiro](/content/@tashapiro/index.html) - [Shirley Wu](/content/@sxywu/index.html) \]\]>https://observablehq.com/blog/anatomy-of-an-impactful-data-visualization-with-karim-douieb https://observablehq.com/blog/anatomy-of-an-impactful-data-visualization-with-karim-douiebWed, 19 Nov 2025 00:00:00 GMTWhen [Karim Douieb](https://bsky.app/profile/karimdouieb.bsky.social) shares a new data visualization, it gains momentum quickly. Over the past decade he has created a number of compelling data visualizations that have introduced new perspectives and sparked thoughtful conversations. We sat down with Karim, data visualization designer and co-founder of [Jetpack.AI](http://jetpack.ai/), to dig into the anatomy of an impactful visualization. Here, Karim shares his views on what resonates with audiences, and how he designs for clarity, impact, and emotion. ## A visual way of thinking Growing up, Karim had been searching for a language that made sense to him. Diagnosed with dyslexia, he struggled to express himself with written words. Drawing, creating schemas, and mapping ideas out visually became his preferred way of thinking and communicating. Early in his career, Karim worked for a company that processed huge amounts of data for advertisers. He found an interest in data analytics, and soon discovered a tool called [Protovis](https://mbostock.github.io/protovis/) — created by Observable founder and co-CEO Mike Bostock — that opened a door to data visualization. “It blew my mind, and made me want to dig deeper. From then on, data visualization became a second language, and allowed me to express myself easily.” ## The anatomy of an impactful visualization: 5 key ingredients With years of experience creating visualizations in both his personal and professional life, Karim has developed an instinct for what resonates with audiences. He has identified five crucial components that make viewers stop scrolling and start talking about his data visualizations: 1. Keep it simple 2. Make it beautiful 3. Add a revelation 4. Make it personal 5. Get a little lucky ### 1\. Keep it simple For Karim, simplicity is the foundation of every successful data visualization. For a visualization to truly have an impact, he says it has to be easily understood by anyone, even those without a particular background. > “When I design something, I imagine whether my grandmother or kids would understand it. They’re the best test audience. If you want your visualization to resonate with the largest number of people, it has to be simple.” _A clever and dynamic animation, this visualization highlights the increase of tornado frequency and magnitude through recent decades._ Getting to “simple” isn’t easy, and often involves a lot of technical effort. Creating more digestible visualizations from complex data requires tools that give you unlimited flexibility. To meet his needs, Karim works with a range of tools like [Observable Plot](/content/plot/index.html), [Observable Framework](/content/framework/index.html), and [Observable Notebooks](/content/platform/notebooks/index.html), and coding languages like JavaScript and React. To keep his work fresh (and to satisfy his own curiosity), he’s frequently adding new tools to his toolkit. His [_All the Passes visualization_](/content/@karimdouieb/all-the-passes/index.html) was created as an experiment using [Three.js](https://threejs.org/), a JavaScript library for 3D graphics and animations. _Karim plotted_ _800,000 passes across 890 major league soccer games played in various major leagues and cups to build a striking_ [_animated visualization_](/content/@karimdouieb/all-the-passes/index.html) _._ ### 2\. Make it beautiful Better data visualizations connect and resonate with more people, and aesthetics play a big role. According to Karim, “People give more attention to something that looks pretty.” _When Karim posted his glossy, 3D_ [_Climate Donut_](https://www.linkedin.com/posts/karim-douieb_dataviz-activity-7378698856269651968-de1o) _on LinkedIn, commenters came out in droves to praise his creation, calling it “creative,” “yummy,” and “eye-catching.” This_ [_visualization_](https://bsky.app/profile/karimdouieb.bsky.social/post/3lzz5zhwfoc2v) _was built on_ [_Observable Desktop_](/content/notebook-kit/desktop/index.html) _, currently in technology preview._ Design decisions, such as color, composition, and motion all play a role in building aesthetically pleasing data visualizations that accurately reflect the data. A well-chosen [color palette](/content/blog/crafting-data-colors/index.html) improves readability and accessibility, while layout and composition make complex information feel more approachable. [Thoughtful animation](/content/blog/effective-animation/index.html) can help make connections and serve as a visual cue, and adding a [variety of chart types](/content/blog/new-advanced-charts-in-observable-canvases/index.html) to a dashboard not only adds richness and appeal, but also exposes patterns and improves understanding. Hooking viewers with a stunning visual means they’re more likely to stick around for the whole data story. ### 3\. Add a revelation An effective data visualization has to give the viewer an “aha” moment. > “When a viewer understands something new from a visualization, or when something clicks for them, they’re more likely to share it.” Karim calls that moment of realization or epiphany an “intellectual revelation,” insisting it’s a key ingredient to creating engagement with his data visualizations. “People want to tell others when they’ve discovered something new.” How does he help his audience get to that lightbulb moment? Transitions and animation — two techniques that show up frequently in Karim’s work. Karim posits that engaging transitions and animations grab viewer attention, and create a sense of drama and discovery. One of the clearest examples of this is a visualization he created in response to a 2016 U.S. election map. “Electoral maps are misleading. Acres don’t vote, people do,” he explained. His approach transformed the traditional land-area map into an animated [cartogram](/content/blog/what-is-a-grid-cartogram/index.html) representing voter results as circles scaled by population. In just a few seconds, viewers grasped how a different visual treatment of the same data could tell a different story. That type of perspective shift is exactly what Karim strives for in his work. ### 4\. Make it personal Karim publishes a lot of visualizations about his hometown of Brussels, Belgium. Over the years, he has come to realize the importance of creating data visualizations that have a personal connection with an audience. In his map, [_Brussels. A Lovely Melting Pot_](https://brussels-diversity.jetpack.ai/), each colored dot represents a person, and each color a community: blue for citizens of the fifteen European member states, pink for North African residents, and yellow for the Turkish population. _Karim developed a data visualization essay that explores the diverse neighborhoods of his hometown, Brussels._ “What I wanted to highlight here is the fact that although Brussels is super diverse, there is an underlying problem in how divided it is as well. These communities aren’t properly integrated into the fabric of the city — they live side by side, but rarely mix. You can really see it with these types of visuals.” While the message of social integration may have struck a chord, Karim suspects the real reason it resonated was simply because people saw themselves in the data. > “People saw it as a love letter to the city. They recognized themselves in it — their neighborhoods, their communities — as opposed to a big, abstract statistic that’s hard to relate to. Each dot could be someone they knew, or even themselves.” _Karim's Baby Names experiment capitalized on the same sense of recognition, with many commenters on_ [_LinkedIn_](https://www.linkedin.com/posts/karim-douieb_babynames-dataviz-typography-activity-7266783232828829696-g8z7/) _sharing screenshots of their own name trending over time._ ### 5\. Get a little lucky Creating a clear, beautiful, and emotionally-resonant data visualization doesn’t always guarantee engagement. Sometimes it just comes down to luck and timing. Karim recalled one particular visualization he was really proud of — an animation of Olympic rings that transformed in size to represent the number of medals won by continent. It didn’t quite get the reception he expected. _Karim's animated visualization highlights the number of rings won by each continent at the Tokyo 2020 Olympic games._ Even with his many years experience and mastery of the craft, he is still surprised by the public reaction when he releases new work. One of his most unexpected viral hits began as a simple experiment. When learning [Blender](https://www.blender.org/), a 3D animation software, he became fascinated by its ability to simulate realistic hair and movement. He then decided to seek out a dataset that he could use to test it out with. “I realized no one had probably ever used hair to display data.” The result was a humorous rendering of a map of the world by male body hair density. It somehow found a fanbase in a niche community on Reddit. _A playful 3D experiment, this visualization of Average Male Body Hair Distribution found unexpected fans on Reddit._ ## Passion — the final ingredient Although Karim’s day job is running his data science company, Jetpack.AI, many of his creations are passion projects fueled by his own curiosity. When asked what he believes unites all of his creations, he answers without pause: > “Looking back, I think what really makes a project work is passion. If I’m not genuinely interested in the topic or the process, it shows. It has to come from personal interest, otherwise, it doesn’t work.” His passion and curiosity push the boundaries of data visualization, which is why he’s known for his striking, creative, often unconventional work. These days, he’s crossing into new territories beyond the digital realm, and exploring sculpture and ceramics as the next frontier. “I’ve been experimenting with ways to make data physical, like using my hands to create sculptures and ceramics,” Karim says. “It’s not directly data-related yet, but I’d love to bring those worlds together. Even if the audience isn’t as enthusiastic as I am yet, it’s a good way for me to express myself.” See all of Karim’s creations in his [portfolio](https://www.karim.news/) and be sure to [follow his public notebooks on Observable](/content/@karimdouieb/index.html). \]\]>https://observablehq.com/blog/clarifying-data-flow-in-observable-canvases https://observablehq.com/blog/clarifying-data-flow-in-observable-canvasesThu, 06 Nov 2025 00:00:00 GMT[Observable Canvases](/content/platform/canvases/index.html) are collaborative whiteboards for fluid data exploration, analysis, and visualization. Across the infinite canvas, charts and data transformations can be freely arranged for flexible exploration that isn’t constrained by traditional linear workflows. This makes it lightweight for data teams and stakeholders to branch off into new analysis paths, and to display findings side-by-side for easy testing and comparison. Clearly representing data flow in such a flexible system is critical. It allows users to track how data is processed, so that findings can be responsibly considered and interpreted. And, it empowers users to confidently update data inputs and outputs so they can compare different slices of the data. But representing data flow isn’t as simple as drawing arrows between nodes. Here, we take you behind the scenes of Observable Canvases to highlight how we’ve designed connectors that clearly visualize data flow, and make it nicer to control. ## **Representing data flow in Observable Canvases** Connectors are visual representations of data flow across a canvas, indicating where data is coming _from_, and where it’s going _to_. Internally, we often refer to connectors as _edges_, which comes from language describing [directed acyclic graphs](https://en.wikipedia.org/wiki/Directed_acyclic_graph), or DAGs. DAGs represent a topological process order, like steps in a sequence. They are composed of _nodes_ (or vertices) representing a state or event, and _edges_ representing paths between them, as shown below. Canvases are DAGs composed of nodes (which can be a Table, Chart, SQL, JS, or Control node) and edges, which are the connectors representing data flow between them. Canvases are both “directed” and “acyclic”: directed, because data is passed from upstream nodes to downstream nodes, and _acyclic_ because data flow can never loop from a downstream node back to a node upstream from it. Below is a canvas with content that maps to the DAG shown above. Connectors are the dashed lines between nodes, with black circles (which we call “ports”) as endpoints. Data: [Open e-commerce 1.0, five years of crowdsourced U.S. Amazon purchase histories with user demographics](https://www.nature.com/articles/s41597-024-03329-6). In canvases, connectors are both a visual cue indicating data flow, and an interactive way for users to manually control data inputs and outputs. By dragging connectors, users can rewire data flow to hop between different exploration paths. A user manually rewires a connector, changing the data input for a downstream chart to explore online purchases across two different regions. Data: [Open e-commerce 1.0, five years of crowdsourced U.S. Amazon purchase histories with user demographics](https://www.nature.com/articles/s41597-024-03329-6). ## **Designing clear connectors amidst complexity** Canvases are built for unconstrained, branching data exploration and analysis. This intentional flexibility means that nodes and connectors can interweave and overlap. Users can freely add or remove content, and easily update data flow. In a poorly designed system, that flexibility could result in a tangled web of content with hard-to-track data flows. But not in canvases! We’ve designed custom connectors to clarify data flow, and make it easier to control. Here are some decisions we made along the way: ### **Defined ports for clear, predictable data flow** One way we simplify data flow in canvases is with defined ports. **Ports** are the black circles indicating where data is passed to or from for any node. There are two ways that we define ports: by location, and by the number of inputs allowed. In terms of location, we restrict _where_ a connector can “attach” to a node. Data inputs always enter on the left side of a node, and outputs exit from the right. We also restrict the number of ports for each node. Currently, the Join node is the only node type with two input ports, since it expects two data sources, as shown below: Defined ports make data flow predictable: users know exactly where to look for a node’s data source and output, and how many inputs are expected. The constraints we placed on ports to simplify data flow are also helpful in avoiding UI conflicts, like overlapping toolbars and connectors that cause user friction. Observable Software Engineer Alex Baldwin explains: > An earlier version of canvases used arrows that could connect anywhere on a node. But while designing other parts of the UI (like the node toolbar), we realized that we didn't have a good way to prevent connectors from colliding with that UI. Having designated input and output ports gives us more control over our UI, and they also make it easier to quickly understand data flow without having to think about arrow directions. Alex Baldwin Observable Software Engineer ### **Guardrails that set you up for success** We aim to make canvases flexible but hard to break. In addition to the visual constraints described above, we’ve put guardrails in place to help users stick to valid data flows, and avoid time wasted on query errors. When it comes to connectors, we do that with smart snapping and restrictions on data flow to avoid loops. Smart snapping is both a convenience feature and a data flow helper. As a user drags a connector near a valid port, it will automatically snap into place as long as it’s an allowed connection (i.e., from an output port, to an input port). Additionally, a connector will turn red, and will not snap into place, if the resulting data flow would create a loop. Below, notice how the connector snaps to nearby output ports, but does _not_ snap to an input port along the way: ### **Zoom handling for a nice UI at any scale** Building an infinitely scalable UI comes with its challenges. We first saw this when ensuring a node’s ports remain well-sized and correctly positioned as a user zooms in and out. Will Chase, an Observable designer who worked on this UI, describes how we address this in canvases: > In an infinite canvas, there's always the question of whether an element should scale with your zoom level, so it appears the same size as you zoom in and out, or whether it should remain a fixed size. For ports, we take a hybrid approach: they scale a subtle amount at intermediate zoom levels, but we clamp the scaling beyond that range so that ports do not appear comically large when zoomed out, or too small when zoomed in. Will Chase Observable UX Designer The resulting ports scale smoothly as a user zooms in and out, keeping connectors visible from a distance, and manageable close up. Custom repositioning and scaling keeps ports in the right place relative to the node, and big enough to see when zoomed out (left) but not distractingly large when zoomed in (right). ## **Explore the leading edge of data analysis** In Observable Canvases, data exploration and analysis isn’t constrained by linear workflows. With this added flexibility, clearly representing data flow is both more challenging, and more critical. Our clear and predictable connectors help users to direct and track data as it transforms in branching analyses across the canvas. Defined ports, helpful guardrails, and refined connector behavior during scrolling and zooming make it easier for users to track and define data flow — and harder to make mistakes. Want to test out Observable Canvases yourself? [Request a demo](/content/platform/canvases/index.html), and get started with a free pilot. \]\]>https://observablehq.com/blog/making-science-visible-dr-philip-bogden-bringing-data-visualization-to-cancer-research https://observablehq.com/blog/making-science-visible-dr-philip-bogden-bringing-data-visualization-to-cancer-researchWed, 05 Nov 2025 00:00:00 GMTSometimes breakthroughs happen where you least expect them — like during a morning row between a data scientist and an oncologist. Out on the water one morning, [Dr. Philip Bogden](https://www.linkedin.com/in/philip-bogden-0501167/) listened as his rowing partner, a friend and practicing oncologist, described the research project he was working on with some colleagues. The research explored how uncertainty shapes cancer patients’ outlooks, testing the idea that patients with poor prognoses find hope in uncertainty, while those with favorable ones respond to it with fear and anxiety. To test their hypothesis, the team had sent out a 30-question survey to collect data from hundreds of patients. Over two hundred responses later, the team found themselves struggling to make sense of the survey data, which was dispersed across a number of Excel spreadsheets. “The data had become too complex, and they had no infrastructure for analyzing or visualizing any of it. No Power BI, no Tableau — just very basic tools in their survey software, REDCap, which weren’t built for exploration,” Dr. Bogden said. That conversation led to an unexpected collaboration and a breakthrough that would change how the oncologists approached their medical research. Dr. Bogden believed that interactive visualization would enable real exploration, streamline communication across the remote team, and spark more meaningful discussions. > “My goal with any visualization project is to facilitate the ability to communicate ideas. I wanted to keep it simple and make sure it worked so that the oncologists could explore their data and identify patterns and relationships more easily.” What began as a simple favor for a friend became a model for using data visualization to reshape medical and scientific collaboration. ## Facilitating exploration with an interactive data app Before he could start building, Dr. Bogden had to make sure the data was ready to work with. Patient health information is extremely sensitive and protected by HIPAA, so the team made sure to mask all personally identifiable information before granting Dr. Bogden access. From there, he tidied the data and got it into a usable format. When it came to tooling, he chose [Observable Framework](/content/framework/index.html) for app development, [Observable Plot](/content/plot/index.html) to create interactive visualizations, and GitHub Pages to publicly host the project so collaborators could access it with a simple link. _Dr. Bogden developed a public-facing data app to share the visualizations with oncology researchers._ Dr. Bogden described the benefit of using Observable with a cross-functional co-working group: > “Data scientists live in Jupyter notebooks, web developers work from the command line, and oncologists use Excel. These communities each have their own tooling, which often prevents them from collaborating. But Observable cuts through all of that and removes the barriers that make it hard for these groups to work together.” With the ability to prototype quickly with Observable, Dr. Bogden was able to get real-time feedback and bridge communication gaps that the group was facing. Framework handled the technical infrastructure needed to build and publish a data app so he could focus on the data and design details. Plot’s concise API made it easy to fine-tune visual elements like axes, labels, and tooltips without getting bogged down in code. He even found time savings from reading through Observable’s documentation, which he described as “outstanding.” All told, that meant he was able to spend less time on app development and debugging, leaving more time for productive discussions about the data. “Together, these tools made the process faster, smoother, and more collaborative. The technologies that Observable offers are genuine game changers.” ## Iteration in action: Building a shared visual language Once he decided on tooling, Dr. Bogden began prototyping. It took plenty of iterations and lots of feedback before the dashboard finally clicked for the researchers — which Dr. Bogden explains is not only common, but critical, to data projects. In the beginning, the researchers told Dr. Bogden that they wanted to “see correlations” in the data. This led him to initially create [scatterplots](/content/@observablehq/plot-scatterplot/index.html), a common chart type used to show correlation. But Dr. Bogden quickly realized that chart type wasn’t going to work. “The language they were using was throwing me off at first. It quickly dawned on me that what they really wanted was to see patterns in the data, not correlation.” He then tried displaying survey responses as faceted [histograms](/content/@observablehq/plot-histogram/index.html). Several iterations later, the researchers decided they preferred summary statistics with grouped [bar charts](/content/@observablehq/plot-bar/index.html), along with the ability to quickly change question groupings and a few key demographic and diagnostic factors. The team also included a pre-med intern whose involvement helped bridge the data and medical worlds. The static charts she prototyped provided inspiration for Dr. Bogden’s interactive visualizations. “The best insights come from showing something, reacting, and improving on it together.” Reflecting on the power of the technology that enabled their iterative process, he said, “That’s the magic of Observable — it makes iterating really fast and easy. What might’ve taken days of back-and-forth is now a live feedback loop.” Faceted histograms vs. grouped bar charts: Feedback and iteration helped the team land on the visualization they found most useful. But it wasn’t until the oncologists started using the app on their own, exploring and interacting with it independently, that he realized he was onto something truly useful for them. > “The difference was when they started diving into the data themselves. That’s when they were able to penetrate the complexity of their own dataset, ask better questions, and start understanding what was going on in the data." ## **Redefining scientific collaboration** Dr. Bogden recalled how the researchers were “amazed” throughout the project — not just by the findings and deliverables, but by the experience of exploring and collaborating through an interactive, HIPAA-compliant visualization tool. “They weren’t used to this level of flexibility — being able to change things, explore, and see the data from different angles. They’d never used an interactive web application like this before, and the fact that it could be public and still HIPAA-compliant really blew them away. It completely changed how they discussed their data.” The team believes that making this, and similar, research accessible to the broader medical and scientific community will have significant positive implications, accelerating new ideas and discoveries. The team is currently drafting a paper about this project and its methods for publication in a top medical journal. For Dr. Bogden, the project reinforced a key belief: “Data visualization is a means to an end. The end is enabling science.” As for the study itself, while some initial findings started to surface, the original hypothesis hasn’t been supported or disproven quite yet. The visualizations have opened the door for pursuing a larger, phase 2 study. What began as a casual conversation between two rowers became an example of how data visualization can bridge disciplines and move science forward, one chart at a time. You can visit Dr. Bogden’s creation for The Uncertainty Project [here](https://pbogden.github.io/uncertainty/). \]\]>https://observablehq.com/blog/eight-features-supporting-fluid-data-exploration-in-observable-canvases https://observablehq.com/blog/eight-features-supporting-fluid-data-exploration-in-observable-canvasesFri, 17 Oct 2025 00:00:00 GMTFast and fluid data exploration allows analysts to focus on what matters most: answering business questions, and discovering deeper insights. With less time spent on routine tasks, analysts have more capacity to investigate new questions and follow emerging patterns. But, too often, data exploration feels cumbersome and time-consuming. This disincentivizes further exploration and leads to shallow analyses that might miss important insights. [Observable Canvases](/content/platform/canvases/index.html) are collaborative whiteboards designed for fast and fluid data wrangling, exploration, and analysis. We’ve written about major features, like [optimized database queries](/content/blog/big-data-fast-data-visualizations/index.html), [transparent AI](/content/blog/playing-safely-with-fire-building-interpretable-ai-for-data-analysis/index.html), and [advanced built-in charts](/content/blog/new-advanced-charts-in-observable-canvases/index.html), that make canvases the best tool for modern business analytics. But we haven’t put a spotlight on the many smaller features that streamline data work in canvases. Read on to learn about eight canvas tips and tricks that speed up data wrangling, keep momentum in your data exploration, and simplify navigation in complex analyses — leaving more time for high-impact analyses and responding to stakeholder requests. ## **Fluid data workflows** Data exploration can reveal patterns and anomalies that inspire new questions and exploration paths. When analysts are constrained by linear workflows that make extra exploration feel like a chore, they’re less likely to pursue those questions, which means they could be leaving some useful findings in the dark. In Observable Canvases, we want to make it easy for analysts to dig into interesting discoveries wherever they pop up during exploration. Here are a few features that make branched, nonlinear data exploration feel lightweight. ### **Duplicate nodes and branches** Data analysts often want to compare different scenarios or slices of their data. Use the option-drag shortcut in a canvas to quickly duplicate one or more nodes. Then, update your new branch to use a different data subset, which you can compare with the original. Below, a user duplicates a branch with chained nodes exploring Amazon purchases for western states. In the new branch, they switch the selected states to the southeast region for a side-by-side comparison for purchases over time, by region. Use Alt-drag or Option-drag to duplicate selected canvas nodes, for a fast way to explore different data scenarios and subsets. Data: [Open e-commerce 1.0, five years of crowdsourced U.S. Amazon purchase histories with user demographics](https://www.nature.com/articles/s41597-024-03329-6). ### **Rewire edges** Edges are the dashed lines connecting two canvas nodes, representing data flow from one node to the next. You can update data flows across the canvas by manually disconnecting edges and reconnecting them to different nodes, for a quick way to test alternate data inputs and outputs. Below, a user drags an edge to switch the chart input from a table containing state-level revenues, to one with regional revenues. Drag edges to control data flow across the canvas, as shown here to update a chart to visualize regional, instead of state-level, revenues. Data: [Open e-commerce 1.0, five years of crowdsourced U.S. Amazon purchase histories with user demographics](https://www.nature.com/articles/s41597-024-03329-6). ### **One click to presentation-ready, interactive charts** Getting a closer look at interesting patterns should be an exciting part of data exploration, not a chore. In canvases, when you see something worth digging into in a summary chart of a Table node, it only takes one click to create a large interactive version. From any column header, select “Create chart” in the dropdown menu to instantly add a large version of the chart downstream on the canvas. Share or embed the new chart as-is, use it as a jumping off point for further exploration, or test different views of the data with other built-in chart types as shown below: Pop out a large, interactive chart from any column header in a Table node in one click, then continue building or explore different views from our [expanding chart library](/content/blog/new-advanced-charts-in-observable-canvases/index.html). Data: [Consumer Complaint Database](https://www.consumerfinance.gov/data-research/consumer-complaints/), Consumer Financial Protection Bureau. ## **Fast data wrangling in the Table node** Data wrangling still takes up a huge chunk of analysts’ time. We want to reduce the time analysts spend on data prep so they can focus on higher-impact tasks like exploration and analysis. In canvases, the Table node provides a preview of data table contents, with concise summary charts that provide a quick glimpse of variable distributions. The Table node doesn’t just help you see your data: it is a data wrangling powerhouse where you can manage columns, filter and sort records, update data types, and more. Here are three Table node features that speed up common data processing tasks. ### **UI-based column management** We recently upleveled column management in the Table node with UI options to rename, select, deselect, and reorder columns. In the video below, a subset of columns are selected and reordered in the Columns pane of a Table node. Note that when the **state** column is renamed atop the column, the new name is automatically reflected in the Columns pane. Manage columns by selecting, excluding, renaming, and reordering columns right in the Table node. Data: [Consumer Complaint Database](https://www.consumerfinance.gov/data-research/consumer-complaints/), Consumer Financial Protection Bureau. ### **Derive new columns** Analysts commonly need to add a new column based on values in an existing column. For example, you might need to calculate an order total based on quantity purchased and unit price. Unit conversions, aggregating groups, and extracting date components (e.g., to get just the year from a date) are other common cases where you’d derive a new column. In canvases, you can now quickly add a derived column right in the Table node. Click the plus sign in the left margin to add a new column. Give it a useful name, then add an expression to populate a new column. Below, a new **complaint\_month** column is added, containing the month number extracted from **date\_received**: Data: [Consumer Complaint Database](https://www.consumerfinance.gov/data-research/consumer-complaints/), Consumer Financial Protection Bureau. ### **Visual filtering in summary charts** The Table node’s summary charts are both concise visual summaries of variable distributions and built-in, interactive visual filters: you can brush over continuous values, or select and deselect categories, for fast visual data filtering right in column headers. When you filter in summary charts, the distributions shown across other columns in the Table node are automatically updated, so you can quickly see how distributions change for different subsets of the data. Table node filters also carry through to downstream nodes, for a quick way to explore patterns by group or subset throughout a chained analysis. In the clip below, a user filters data by first brushing over a temporal histogram of orders over time, then selects two age bins, then deselects two education categories. Note how both the other columns in the table, as well as downstream nodes, update to reflect the new subset. ## **Easy canvas organization and navigation** Fluid data flows and branching paths are great for richer and more thorough data exploration, but can also quickly produce a large network of interconnected nodes that are difficult to keep track of. Here are two features that help you keep things tidy and trackable in canvases. ### **Navigate with the minimap** We see canvas sprawl as a good thing: it’s a signal that you’re conducting robust visual exploration and testing. As your analysis grows, use the minimap — a zoomed out view of canvas content — to see and navigate around your work. Selected nodes are highlighted in the minimap, serving as useful waypoints. Below, a user drags the focus area (indicating what’s currently in view) across the minimap to find and return to a specific chart node. The minimap provides an interactive view of the entire canvas, for easier navigation as your analysis grows. Data: [Open e-commerce 1.0, five years of crowdsourced U.S. Amazon purchase histories with user demographics](https://www.nature.com/articles/s41597-024-03329-6). ### **Minimize nodes for a simplified view** In Observable Canvases we aim to keep your eyes on the data at every step. [Visual data analysis](/content/blog/visual-data-analysis/index.html) helps analysts and stakeholders to notice unexpected patterns, catch mistakes sooner, and responsibly interpret analyses. But we get it: that approach means displaying every node by default, which can be visually distracting. For a pared-down view, minimize any node by clicking on the minimize icon in the top bar, choosing “Minimize node” in the three dot menu, or using the Command-M shortcut. So you don’t miss out on important data transformations, we indicate if one or more filters are applied in a minimized node, as shown below: ## **We’re building fast so you can, too** In this post, we’ve highlighted eight of the many Observable Canvas features that improve data workflows, accelerate data wrangling, and ease canvas navigation. The table node lets you quickly add, rename, select, and reorder columns — along with interactive filters and the option to pop out instant large charts from any column header. Rewire connections between nodes and duplicate branches for lightweight exploration and testing. As your work expands across the infinite canvas, simplify chained analyses by minimizing nodes and easily navigate content using the minimap. We’re continuously adding new features and functionality in canvases that improve how analysts and stakeholders can explore, analyze, and display data. To try it out for yourself in a free pilot, [request a demo](/content/platform/canvases/index.html) today. \]\]>https://observablehq.com/blog/dashboards-in-observable-canvases https://observablehq.com/blog/dashboards-in-observable-canvasesTue, 07 Oct 2025 00:00:00 GMT“Dashboards are dead!” If your work involves data analysis and reporting, you’ve probably seen this headline pop up in your social media feed. Unfortunately, the claim hits close to home for many BI analysts and stakeholders. [Many dashboards rot into obscurity](/content/blog/five-strategies-fight-dashboard-rot/index.html) because they fail to deliver on their promise to improve data exploration and business decision-making. Why? Shallow insights, static data visualizations, and unclear takeaways lead to dwindling user value and engagement. Charts break, and dashboard content becomes outdated or unusable. Upkeep gets put on the backburner as new tasks take priority. As a result, analysts’ hard-earned dashboards end up decaying in a crowded data app graveyard. But dashboards aren’t doomed to fail. When done right, they serve as the go-to place where teams explore data and track essential metrics. Successful dashboards can be a hub for data-driven business discussions and decisions. In Observable Canvases, we’re addressing the most common causes of dashboard rot — and we’re moving fast. We recently rolled out major updates to canvases including a new dashboard viewer experience, new control nodes that make it lightweight to add custom interactions to your dashboards and embeds, and a dashboard builder for streamlined development. Read on to learn how canvases: - Keep data exploration, analysis, and dashboarding in one place for lightweight development and upkeep - Consolidate collaboration for precise, in-product discussions that minimize crossed wires and back-and-forths - Put expressive, interactive charts at your fingertips, plus the option to build with code for full customization ## **Data exploration, analysis, and dashboard development in one place** In canvases, dashboards are a natural extension of your data exploration and analysis, rather than the output of a separate, additional workflow. Our all-in-one approach keeps every step of the data analysis process — from database access, to data exploration and analysis, to dashboard development — in the same place. This avoids costly tool switching, limits redundant work, and makes it easier to keep dashboards fresh as your team tunes and expands your analysis. Key to streamlined development is our new dashboard builder, which lets you quickly create a polished dashboard directly from existing canvas content. Just choose which nodes (charts, tables, controls, big numbers, etc.) you want to include in the dashboard. Those components are aligned to a grid, and responsively resized, for an organized view that’s immediately ready to share with teammates and stakeholders. In Observable Canvases you can create a ready-to-share, interactive dashboards directly from selected nodes. When data updates, or when you make edits to charts or analysis in the canvas, the dashboard refreshes automatically to ensure that your users are always seeing up-to-date information. ## **Modern collaboration meets BI** Cross-functional collaboration is key to successful dashboard development and sustained use. Stakeholder input helps analysts know what’s most useful for end users, and keeps dashboards relevant as business questions and priorities change. With legacy BI tools, collaboration can be disjointed. Analysts and stakeholders often resort to communicating across different channels, like Slack or email, at a distance from the actual dashboard they’re discussing. This makes it difficult for users to pinpoint feedback on a specific part of an analysis, resulting in excessive back-and-forths and crossed wires. And, when users aren’t directly engaged in analysis and dashboard development, there’s less buy-in and trust in the final product. Collaborating across tools and at a distance from dashboard development leads to more back-and-forths and crossed wires between data analysts and stakeholders, which wastes time and stifles momentum. [Observable Canvases](/content/platform/canvases/index.html) are collaborative whiteboards that bring data analysts and stakeholders together to participate in analysis and dashboard development in the same place. Stakeholders can join analysts directly in a canvas, using built-in tools like text annotations, highlighting, manual illustration, and sticky notes to leave unambiguous feedback directly on drafted charts and dashboard content. This avoids dropping a complete dashboard draft into a stakeholder’s lap for review, which can leave them feeling less invested in the process, less empowered to suggest big changes late in the game, and less confident in their interpretation of dashboard content since they’ve been detached from the analysis and discussions along the way. Canvases are built for [collaborative analytics](/content/blog/collaborative-analytics/index.html), allowing data analysts and stakeholders to meaningfully work together from ideation to publication for greater dashboard value, understanding, and user buy-in. Collaboration happens directly within a canvas alongside your analysis and dashboard development. Whiteboarding tools help stakeholders engage directly, using highlighting, text, sticky notes, and drawing for fast, clear discussions. ## **Build expressive, interactive charts with UI or code** Some legacy BI tools force data analysts to choose between fast, UI-based chart development and rich, interactive visualizations. Constraints on chart types and customization frustrate dashboard developers, and result in generic displays that can’t capture the richness or nuance in complex business data. In canvases, we aim to deliver the best of both worlds: the quickest way to generate state-of-the-art, interactive charts with our no-code UI, plus the option to build fully custom visuals in JavaScript. ### **Expressive charts at your fingertips** Our built-in chart library lets you build expressive charts with just a few clicks. From essentials like line and bar charts, to [advanced visualizations](/content/blog/new-advanced-charts-in-observable-canvases/index.html) like Sankey diagrams, arc maps, and beeswarms, you can create and toggle between different views of your data in minutes. Every chart in canvases is optimized for performance and clarity. Smart queries return [speedy visuals even when you’re working with huge data](/content/blog/big-data-fast-data-visualizations/index.html), and are designed to ensure that outliers and noise don’t drown out meaningful patterns. All charts are designed to be presentation-ready out of the box, and we’re continuously adding more layout and styling options for quick customization. When it comes to interactivity, our charts are shipped “batteries included.” Chart interactions (like brushing, tooltips, multi-select, and deselect) are included by default and persist once the chart is added to a dashboard, allowing viewers to explore different slices of the data with no additional development needed. If you’re looking for custom interactions beyond what comes with our built-in charts, we’ve got you covered! Our new control nodes give you a low-lift way to add interactive widgets like radio buttons, value sliders, and date ranges for even more versatile exploration in dashboards and embeds. And don’t worry — while updates you make in the canvas are reflected in the dashboard, it’s not a two-way street. User interactions with charts or control nodes on the dashboard don’t affect your canvas, so your work and visualizations stay intact behind the scenes. ### **Or, build custom charts with code** If our chart library doesn’t have exactly what you need, you can build custom visualizations with code in JavaScript nodes. Then, add your creation to the dashboard right alongside our built-in charts and tables. The option to work interchangeably between UI, AI, and code across the canvas means you’re not constrained to our built-in charts. And, you don’t have to deal with tedious tool switching and integration issues in order to build exactly the visualization you want for your dashboard. You’re not limited to our built-in chart options in canvases. You can choose between using code, UI, and AI at any point, including to create custom data visualizations in JavaScript nodes. ## **Dashboards are alive and well in Observable Canvases** With Observable Canvases, we’re addressing the most common dashboarding pain points for creators and viewers. New features including our improved viewer experience, UI dashboard builder, and control nodes, alongside our growing library of interactive charts, give analysts a faster path from messy data exploration to presentation-ready display in one place. Our recent updates also improve the viewer experience, providing more ways to interact with and explore visualizations while keeping data fresh. We have exciting plans for more dashboarding features and management options just around the corner, and will keep you posted as they roll out over the next month. Want to see how canvases streamline data analysis and dashboarding in a single workflow, consolidate collaboration, and deliver expressive charts to your users? Observable Canvases are now in general availability. To learn more, and to test it out yourself in a free pilot, [request a demo today](/content/platform/canvases?modal=demo/index.html). \]\]>https://observablehq.com/blog/right-chart-for-data https://observablehq.com/blog/right-chart-for-dataFri, 19 Sep 2025 00:00:00 GMTCreating a data visualization seems simple enough, until you realize how a decision like selecting a chart can impact your ability to communicate insights effectively. Selecting the right type of chart, and making smart design decisions along the way, is crucial for both clarity and impact because the chart type needs to align with the type of data you are displaying. In this post, we’ll break down which charts work best for different types of data, including time series, categorical, hierarchical, and more. ## **The best charts for time series data** [Time series data](/content/blog/analyzing-time-series-data/index.html) consists of values recorded at successive points in time and ordered by the time when they were recorded. It's commonly used for tracking how things change over time — like sales, website traffic, or temperature — and can help teams uncover trends, spot patterns, and make future predictions. ### **Line charts** Line charts, which visualize quantitative or temporal observations by interpolating between adjacent data points, are common choices for time series data. A line chart visualizing the change of Apple’s stock price over time. Most data visualization libraries connect records in [line charts](/content/learn/glossary#line-chart/index.html) with a straight line from point-to-point by default. For high-frequency time series data collected at predictable intervals, this is often the best choice. However, there may be scenarios where you want to change how the line is drawn between points, such as if you want to visually indicate stepwise changes. Try it for yourself: To build your own line chart using [Observable Plot](/content/plot/index.html), simply copy the below example snippet into a JavaScript cell in an [Observable Notebook](/content/platform/notebooks/index.html), or select “Line chart” from the cell menu. Connect your data, and update the data and variable names to match what’s in your data. Plot will automatically generate a time series line chart handling scales, axes, and tooltips. ``` Plot.plot({ marks: [\ Plot.ruleY([0]),\ Plot.lineY(aapl, { x: "Date", y: "Close", tip: true })\ ] }) ``` Plot also offers over 20 curve options, which determines how values between points are interpolated. To update your chart using any of the built-in curves, simply add your preferred curve type as an option within your line mark. ### **Sparklines** For a compact alternative to traditional line charts, consider sparklines. Sparklines are miniature charts that are ideal for showing trends across multiple categories in tight spaces. Since they often omit axes and labels, they're best used when the goal is to highlight general patterns. ### **Bar charts** While line charts are common, they’re not always the most effective choice. Bar charts are typically used for categorical domains, but they can also visualize aggregated data over time, such as a company’s total month-to-month or quarterly sales by binning the data. Plus, [bar charts](/content/learn/glossary#bar-chart/index.html) can be more legible than line charts when there are fewer values to display. To go deeper into the decision about when to choose a line chart or a bar chart, check out important considerations [in this blog](/content/blog/bars-vs-lines-time-series-data/index.html). ## **The best charts for categorical data** ### **Bar charts** As mentioned above, bar charts are most frequently used to visualize categorical data by displaying numeric values grouped by category. For example, you might use a bar chart to show product engagement by user profile, or the number of churned customers by subscription type. The length of each bar represents a numeric value, which can come from two sources: 1. An pre-calculated value by group, like a sum, count, or mean, for a numeric column in the data, such as "product visits." 2. A count of how many records fall into a given category, like the number of tickets labeled "technical issue." The categorical axis organizes the data into distinct groups that can be ordered in whatever arrangement will be most useful to the viewer. For example, the bars may be sorted by the value that's represented in either ascending or descending order, or they may be sorted by the labels on the categories. _Bar chart showing the distribution of letter frequencies in an English-language corpus._ In some cases, a horizontal bar chart is preferred, for example to avoid overlapping x-axis labels when you have long category names. There are many variations on the bar chart, including the [stacked bar chart](/content/@observablehq/plot-stacked-bar-chart/index.html) (which we describe in greater depth below) to show the breakdown of values by subcategories within each group. See more bar charts examples, including a diverging bar chart and a grouped bar chart, in the [Observable Plot gallery](/content/@observablehq/plot-gallery#bar/index.html). ### **Radar charts** Radar charts are polar line charts that represent categorical data. While they may look interesting, radar charts should generally be avoided because the order of data dimensions can impact how the data is interpreted. [Learn more](/content/blog/avoid-radar-charts/index.html) about why radar charts are usually not the best choice. A radar chart showing the results of a survey of smartphone buyers. ## **The best charts for part-to-whole relationships** When you want to compare how individual components relate to the overall total, you should reach for visualizations that display the size of each part relative to the whole, how each part contributes to the total, and how the whole value is divided across categories. ### **Stacked bar charts** Stacked bar charts add a fill color or style to each bar that differentiates subcategories within each group. Stacked bars can show absolute values like counts or sums for each subcategory, or stacked values can be normalized to more clearly show proportions. For example, the normalized stacked bar chart below, which uses [stack options](/content/plot/transforms/stack#stack-options/index.html) in Observable Plot to normalize values, shows the proportions of U.S. dams with low, significant, or high risk classification between different dam types: Here's the Observable Plot code we used to create this stacked bar chart: ``` Plot.plot({ marginLeft: 180, x: { percent: true, label: "Percent" }, color: { legend: true, domain: ["Low", "Significant", "High"], range: ["#70B362", "#FFE54A", "#FF704D"] }, marks: [\ Plot.barX(\ dams,\ Plot.groupY(\ { x: "count" },\ {\ filter: (d) => d["Hazard Potential Classification"] != "Undetermined",\ y: "Primary Dam Type",\ fill: "Hazard Potential Classification",\ offset: "normalize"\ }\ )\ ),\ Plot.ruleX([0])\ ] }) ``` ### **Pie charts** While commonly used, pie charts are often criticized within data visualization circles due to a perception that they can be difficult to accurately interpret. [Pie charts](/content/learn/glossary#pie-chart/index.html) work best with the number of parts (slices in the pie) are fewer, when there are clear differences in proportions by section, and when the different parts add up to 100%. Donut charts are a common variant of the pie chart without the center. Donut charts are sometimes used for a better data-ink ratio, or when analysts want to add an additional summary value in the center. Get a primer on why pie charts are controversial in [this post](/content/blog/truth-about-pie-charts/index.html). ### **Waffle charts** Waffle charts represent parts of a whole in a digestible, grid format that allows viewers to more easily infer exact proportions. In waffle charts, values are broken down into equally sized cells, often squares, that can be quickly counted and compared across groups. For example, the waffle chart below shows the proportions of U.S. dams with low, significant, or high risk potential classifications for the top five most common dam categories with each circle representing ten dams. Data: U.S. National Inventory of Dams You can quickly make waffle charts in Observable Plot using the [waffle mark](/content/plot/marks/waffle/index.html). Customize your chart with built-in options for units (the counts or amount each cell represents), gap size between cells, value rounding to avoid partial shapes, and more. ## **The best charts for hierarchical data** Hierarchical data represents parent-child relationships. ### **Treemaps** [A treemap](/content/blog/treemaps-history/index.html) subdivides space to show the proportions of a value as part of the whole for hierarchical data. For example, to visualize sales by department and product, the area of each rectangle would show the amount of sales for each, as a fraction of the total sales shown in the chart. Unlike some other chart types, treemaps are also good for [packing information in small spaces](/content/blog/big-insights-small-spaces/index.html). In the example treemap of orders in our [Pizza Paradise demo dataset](/content/@observablehq/pizza-paradise-data/index.html), color represents the pizza category, and each category is further subdivided by pizza type. Rectangle size represents the number of orders for each. This example treemap is relatively basic; to see more complex variants, such as [cascaded](/content/@d3/cascaded-treemap/index.html) and [nested](/content/@d3/nested-treemap?intent=fork/index.html) treemaps, visit the [D3 gallery](/content/@d3/gallery/index.html). ### **Circle packing charts** A [circle packing chart](/content/@d3/pack/2/index.html) is an alternative to a treemap that visualizes hierarchical data as nested circles, with circle size typically reflecting a quantitative value (such as a count or sum) for each group. While circle packing charts use space less efficiently than treemaps, they can sometimes make it easier for your user to interpret hierarchy. If this chart type is new to you, be sure to check out pro tips for creating circle packing charts in this [Observable Notebook](/content/@didoesdigital/about-circle-packing/index.html). Data: Consumer Financial Protection Bureau [Consumer Complaints Database](https://www.consumerfinance.gov/data-research/consumer-complaints/) ### **Sunburst charts** Another type of chart that works well for hierarchical data is the [sunburst chart](/content/@d3/sunburst/2/index.html), which consists of a series of concentric rings that correspond to a level in the hierarchy. Each ring is divided into slices, which can be sized equally or proportionally, depending on an associated value. Sunburst charts can be static, or they can be interactive. For instance, this [zoomable sunburst](/content/@d3/zoomable-sunburst/index.html) displays just two layers of hierarchy at once, with the ability to zoom in or out for more detail. A slightly different take on a sunburst is the [icicle diagram](/content/@d3/icicle/2/index.html). ## **The best charts for geospatial data** There are many ways to visualize geospatial data, but here are five different map types to get you started. ### **Choropleth maps** A choropleth map uses colors, shades, or patterns to show how a data variable changes across different geographic regions. They help reveal patterns or differences in the data by using color progressions — such as light to dark or one hue to another — to represent varying values. One consideration to keep in mind when building choropleths is that they can make it difficult to read exact values and larger areas can appear more important than smaller ones (even if the data doesn’t support that). To get started creating your own choropleth, check out this [Observable Plot tutorial](/content/@observablehq/plot-bivariate-choropleth-explained/index.html). ### **Bubble maps** To show magnitude and compare proportions across geographic regions, you may want to choose a bubble map. Estimated population by county, 2016 Similar to a bubble map, a [spike map](/content/@observablehq/plot-spike/index.html) uses vertical spikes to represent quantitative values, with the spike height corresponding to a specific data value. Instead of using area to encode data, like bubble maps, the spike height can reduce overlap in dense areas, making them a good choice for datasets like population density. ### **Arc maps** An arc map visualizes flows or connections between locations. In the example below, the map shows seafood imports to the United States, with arc thickness and color dependent on import volume from each exporting country (in kilograms). Data: NOAA [US Trade in Fishery Products](https://www.fisheries.noaa.gov/foss/f?p=215:2:1986023721187) ### **Grid cartograms** A [grid cartogram](/content/blog/what-is-a-grid-cartogram/index.html) represents subregions using tiles of the same shape, arranged to reflect their general geographic locations while avoiding overlap. The most common tile shapes are squares and hexagons, but they can also be [rectangle cartograms](/content/@neocartocnrs/rectangle-cartogram/index.html), [dot cartograms](/content/@neocartocnrs/dots/index.html), and [more](/content/blog/what-is-a-grid-cartogram#what-are-some-common-alternatives-to-grid-cartogra/index.html). The biggest drawback to grid cartograms is that they aren’t geographically accurate representations. But, [they have numerous benefits](/content/blog/what-is-a-grid-cartogram#what-are-some-of-the-pros-and-cons-of-grid-cartogr/index.html) — such as helping viewers compare values between regions and creating equal representation between smaller and larger regions. If these examples have sparked a deeper curiosity about map-making, then you won’t want to miss [this interview with data visualization developer Fil Rivière](/content/blog/maps-and-data-visualization-with-fil-riviere/index.html) and this [video about using dynamic cartograms](https://www.youtube.com/watch?v=1dxP8DK9jHI) to visualize election data. ## The best c **harts for distribution of values** ### **Histograms** One of the most useful data visualization types for demonstrating how values are distributed, and the shape of that distribution is the [histogram](/content/blog/simplicity-bar-charts#histograms-the-one-dimensional-bar-chart/index.html). In histograms, data is grouped into bins that display the count of how many data points are in each bin. ### **Beeswarms** A visually interesting alternative to other charts that show a distribution of values is the beeswarm, which visualizes observations for a single quantitative variable and uses jitter to avoid overlapping marks. Encodings for color and size can represent additional variables in the chart. The beeswarm chart below visualizes the distribution of product price by category. Data: [Amazon e-commerce data](https://www.nature.com/articles/s41597-024-03329-6) _Data:_ [_Amazon e-commerce data_](https://www.nature.com/articles/s41597-024-03329-6) ### **Box plots** Another alternative to a histogram for visualizing distribution of values is the box plot, sometimes called a box-and-whisker plot, which summarizes quantitative distribution using boxes and lines (“whiskers”) to visualize the spread of the data. In this [example](/content/@d3/box-plot/2/index.html) from the D3 gallery, the price distribution (y-axis) of a set of diamonds is plotted for a given range of carat values (x-axis). ## **The best charts for showing relationships between variables** ### **Scatter plots** To visualize correlations or patterns between two numeric variables, data analysts will oftentimes choose a scatter plot. In scatter plots, data is shown on two continuous axes with each data point plotted so that its horizontal and vertical positions represent the values of the two chosen data dimensions. Color or symbols can also be used to encode specific variables as is done, for instance, in [this example](/content/@observablehq/plot-symbol-channel/index.html) that visualizes flipper length and body mass for different types of penguins. Data: [Palmer archipelago penguins data](https://journal.r-project.org/articles/RJ-2022-020/) ## **The best charts for showing changes in rankings** ### **Bump charts** Bump charts show changes in rankings by group, usually over time. For example, a bump chart might visualize shifts in product rankings by annual volume sold, or changes in how survey participants rate the importance of issues (like economy, environment, education, healthcare, or foreign policy) when considering political candidates. In the bump chart below, we can track the rankings for selected U.S. seafood imports over time by product: Data: NOAA [US Trade in Fishery Products](https://www.fisheries.noaa.gov/foss/f?p=215:2:1986023721187) ## **Conclusion** Selecting the right chart type for your data is a fundamental part of data exploration, analysis, and communication. Once you’ve selected the best chart for your dataset, you can start evaluating whether or not to add additional elements, like [brushing](/content/blog/linked-brushing/index.html), [animations](/content/blog/effective-animation/index.html), or [interactivity](/content/blog/how-to-build-dashboards-observable-framework-plot/index.html). To start building the more advanced chart types covered in this post, explore and fork notebooks from our D3 and Plot galleries: - [D3 gallery](/content/@d3/gallery/index.html) - [Plot gallery](/content/@observablehq/plot-gallery/index.html) \]\]>https://observablehq.com/blog/table-virtualization-in-observable-canvases https://observablehq.com/blog/table-virtualization-in-observable-canvasesThu, 18 Sep 2025 00:00:00 GMTData analysts need to explore and answer questions using big company data, often with short turnaround times to deliver on manager or stakeholder requests. But when data tools get overwhelmed by large datasets, analysis can be interrupted by spinning wheels, delayed interactions, and painstakingly slow charts and tables that discourage experimentation. At Observable, we aim to make your data work feel fast and fluid, no matter the size of your data. We also want to [keep data visible](/content/blog/visual-data-analysis/index.html) throughout analysis so that you can track changes and patterns in your data every step of the way. That’s why every node in [Observable Canvases](/content/platform/canvases/index.html) provides a window into your data, quickly displaying large underlying datasets as interactive tables, concise summary charts, or larger visualizations. Canvases also encourage branched, collaborative exploration that can result in a great number of interlinked nodes on the screen at any moment. This poses a challenge, because those two goals — great performance, and pervasive data visualizations — are somewhat at odds with each other: the more data you try to _show_ in the browser, the more you’re going to bog it down. One way we’re able to maintain both speed and data visibility in canvases is by virtualizing tables. But what is table virtualization, and how does it improve app performance? Read on to learn the basics of table virtualization, how it impacts canvas speed and user experience, and some unique challenges we overcame during implementation. ## **What is table virtualization?** When exploring data, you probably don’t want to view it all at once. For example, viewing all 80 columns and 6M rows of a table at the same time is (1) practically impossible, and (2) makes things slow and unmanageable. That’s why most tools, including Observable Canvases, encourage viewers to view a smaller subsection of a table, then allow a user to scroll or expand to see more. For example, canvases show the first six columns and eight rows by default: The question then becomes: if you’re onlyever seeing a very small subset of rows and columns at any given time, does the browser really need to render _all_ columns and rows in a table? Spoiler alert: No, it doesn’t. Enter table virtualization. Table virtualization is a “what you see is what **the page** gets” approach to reducing how much content gets rendered on a web page at any time. It’s useful because rendering a large number of DOM elements can overwhelm the browser, making pages sluggish and frustrating to use. **What’s the DOM?** DOM stands for Document Object Model. The DOM is how your browser represents a web page’s HTML source code so it can render elements like paragraphs, images, and table rows or columns on the page. By virtualizing tables, you limit what gets added to a page at any time by only rendering rows and columns that are currently visible to a user. As a user scrolls to a different section of the table, the previously rendered DOM elements that have been pushed outside the visible area are recycled, and repurposed in new positions to represent the table cells now in view. In some cases, buffer columns and rows just beyond the visible area are also pre-rendered to ensure a smooth experience when a user scrolls through or expands a table. Since columns and rows around the edges are already waiting in the wings, they appear instantly. If that wasn’t the case, a viewer could see those elements being rendered on the page as they scrolled into view. Here is a representative illustration comparing virtualized and unvirtualized tables: ## **How does table virtualization impact canvas performance?** Table virtualization drastically improves canvas performance and user experience. In canvases, we want users to pursue nonlinear, branched data explorations with rich tables and data visualizations displayed throughout. Canvases also support multiplayer editing, which means even _more_ content is added as collaborators follow new analysis paths. Observable Canvases encourage branching data exploration and the pursuit of new analysis paths, which means they often end up with a dense network of interconnected nodes. The ease with which users can add new charts and tables, combined with our bias toward data visibility (for example, by automatically showing a table preview with summary charts for each operation), means that a huge amount of data can underlie what’s visible on the canvas at any moment. Without virtualization, all of those out-of-view columns, rows, and summary chart SVGs are rendered unnecessarily. This quickly overwhelms the browser, severely degrading canvas speed and usability. Michael Cooper, Observable Software Engineer, describes why table virtualization is a game changer: > Even when you just have a handful of nodes on the screen, each with something like a thousand cells, that adds up very quickly in terms of browser workload. Without table virtualization, if you try to zoom in or pan around the canvas, it just crawls. And with more complex canvases, or when data has a large number of columns, it can crash the browser. Virtualization is the difference between the app feeling smooth, or feeling janky and awful. Michael Cooper Observable Software Engineer Table virtualization isn’t just important when working with really big data. Even with smaller tables, it can make a difference. “The more data, the more important table virtualization is,” Michael explains. “But you start seeing benefits anytime you have more data in a node than you can physically see at once on the screen.” ## **What unique challenges came up when implementing table virtualization in canvases?** The collaborative, interactive nature of tables in canvases posed unique challenges when implementing table virtualization. The first challenge relates to how we deal with different collaborators who are interacting with the same table, at the same time, from different devices. Those users may be independently scrolling through tables to explore different slices of the data. Michael explains how this complicates table virtualization: > Just because one user can only see some of the columns, doesn't mean that all other users see the exact same thing. So we had to make sure that virtualization is tied to each user’s specific instance of the canvas, and not the general multiplayer instance. Michael Cooper Observable Software Engineer By applying table virtualization to each user’s canvas instance, we ensure that teammates and stakeholders can independently explore tables, while cutting down everyone’s browser workload. A second challenge relates to keeping table contents in sync. We use query snapshots to store the complete data output from each query. Snapshots ensure that table elements don’t ever exist in a Frankenstein-esque state, where outputs of a previous and current query are temporarily intermixed as the updated values are rendered. Previously, when we ran a query in a virtualized table, the invisible (unrendered) columns were ignored. But then, when a user scrolled right or left, summary charts atop table columns would run their SQL queries the moment before coming into view. That would lead to users seeing newer summary charts load and appear alongside the older ones from the last snapshot. To avoid this, we needed to ensure that all columns — not just the currently visible ones — were updated to use the new query snapshot.Michael summarizes: “We had to level up our querying game to understand that sometimes you still need to run queries, even for the things you can't see.” ## **Dive into canvases with the Observable team** Table virtualization is one way we’re making canvases a fast, fluid tool for data analysis and visualization. By only rendering visible columns and rows, we reduce how much of your browser’s brainpower is spent rendering unseen elements, so more is leftover to support smooth user navigation and interactions. Learn how Observable streamlines data analysis, gets you from raw data to state-of-the-art interactive data visualizations in minutes, and brings data teams and stakeholders together to align on business decisions. [Request a demo today](/content/platform/canvases/index.html). * * * ### **Post script** In an early draft of this post, I had written that “viewing all 80 columns and 6M rows of a table at the same time is impossible.” I was corrected by Michael, who commented: “Fun fact: if you had a monitor that was 27 kilometers tall you could show all those rows.” 😆 Here is the record of his calculation, which assumes 135 DPI and a ~24 px minimum row height: It now reads “practically impossible.” \]\]>https://observablehq.com/blog/data-joins-in-observable-canvases https://observablehq.com/blog/data-joins-in-observable-canvasesThu, 04 Sep 2025 00:00:00 GMTMost companies store their data in databases that house a large number of related tables. To answer business questions using that data, data analysts combine records from different tables using [joins](https://learn.microsoft.com/en-us/sql/relational-databases/performance/joins?view=sql-server-ver17) to consolidate variables needed for their analyses. That makes joins one of the most essential and frequently used operations in all of data wrangling. Given joins’ prevalence in data work, we prioritized joins as one of the first operations needed in [Observable Canvases](/content/platform/canvases/index.html) — our browser-based collaborative whiteboards for data analysis and visualization. Building out the UI for such a common data transformation should be straightforward…right? Not quite. It turns out that implementing UI-based joins is challenging. To learn why joins are surprisingly hard to get right and how we handle them in canvases, I spoke with Observable Software Engineer Tony Sullivan. Read on to learn why the engineering work needed to deliver fast, friendly joins in canvases was less than straightforward, and how we made it happen! ## **Avoiding SELECT \*** Observable Canvases work by sending optimized queries to a connected database. Those queries are constructed based on user selections and changes to the canvas. So, you might imagine a UI where a user simply selects two tables ( _items_ and _prices_), indicates a column to join on, clicks a “Join” button, then the following query is run to return all fields from both tables using a full join: ``` SELECT * FROM items i FULL JOIN prices p ON i.id = p.item_id ``` That may seem simple enough. But unfortunately, this query doesn’t work well as a general approach to UI-based joins for several reasons. First, for company databases where analysts have access to some but not all fields (also called [column-level security](https://docs.snowflake.com/en/user-guide/security-column-intro), or column-level access control), using  `SELECT *` would return an error because the query requests columns that a user does not have permission to access. Second, problems can emerge when joining tables that contain identically-named columns. This can result in an error, or in some cases, with auto-generated column names (e.g. _state_ and _state\_1_ for a variable named “state” returned from a left and right table in the join, respectively). For these reasons, we avoid using `SELECT *` in all canvas queries. We take the harder (but more robust) approach of constructing queries that explicitly point to columns in both tables by name. In canvases, when a user selects two tables to join, we need to know two things: (1) which column names to include in the `SELECT` statement, and (2) in the case of duplicate column names, how to alias them. In other (SQL) words, we want to build a query that includes a `SELECT` statement looking more like this: ``` SELECT   i.id,   i.name,   i.category as category_left,   p.price,   p.discount,   p.category as category_right FROM items i JOIN prices p ON i.id = p.item_id ... ``` Aliasing is built into the query to de-duplicate matching column names (here, to return category\_left and category\_right, since a column named **category** exists in both tables). That query doesn’t look too complex. So why does building it from a user’s UI selections pose a challenge in canvases? The answer, in a nutshell, is because joins might be done at any point in the data analysis process. By the time a user chooses to join tables in a canvas, they may have wrangled the data — selecting or deselecting columns, renaming things, deriving new fields, etc. — in ways that complicate the process of building queries that run successfully against the _original_ database. Read on to learn how we construct queries to join tables amidst other data transformations and on a changing, interactive canvas. ## **Handling changes in a reactive canvas** Canvases are reactive. Changes you make to data anywhere on the canvas — like filtering rows, selecting columns, grouping values by category, etc. — are automatically propagated through to downstream nodes so your work stays in sync. When building canvas nodes, we considered how to keep reactivity efficient. “To keep things fast,” Tony shares, “we want the canvas to be synchronous. When something changes upstream in a canvas, we want every node to know what columns to request from the database so the query isn’t delayed. That way, you don't see canvas updates walk their way through downstream nodes one at a time.” For that to happen, each node needs to update automatically when any node upstream is changed. Each node receives its parent node’s query, wrapped in a [common table expression](https://learn.microsoft.com/en-us/sql/t-sql/queries/with-common-table-expression-transact-sql?view=sql-server-ver17), or CTE. As canvases grow in size and complexity, the queries constructed from input CTEs and the current node operations become quite involved — even when a node has a single input! Joins compound this further because they have to watch and respond to _two different_ inputs. Here's Observable Software Engineer Tony Sullivan on why this poses a challenge: > With joins, we have two inputs — the two different tables that are coming into the join node. That node has to look at all of the column names and column types that were computed up the chain, through every single operation in the canvas. Then inside of join, we have to specify which tables are left and right, join them based on a field you picked, then figure out the full list of columns and de-duplicate before sending the query to the database. Tony Sullivan Observable Software Engineer One caveat here: if upstream changes invalidate any of the join inputs (the two connected tables, and the variable they’re joined on), we want to show users a helpful error message before the misconfigured query is ever sent to the database. For example, in the video below, the CATEGORY column used in the Join node downstream is renamed to TYPE. Instead of assuming that the name change should automatically carry through to the Join node, a warning icon appears and the join query isn’t attempted, since CATEGORY no longer exists in the left table. After correcting the join variable to the new variable name (TYPE), the query is successfully re-run. Tony describes our thinking behind this approach: > If upstream you change the name of a column used in the join, the operation should recognize that the column is no longer there and show you the list of the new columns (including the one you renamed). It shouldn’t try to rerun the query because it knows it's not valid. In general, we are trying not to magically fix things for users. We just try to make it really clear if the join (or our other operations) experience an issue that requires changes from the user. Tony Sullivan Observable Software Engineer ## **Extra design considerations for the Join node** Joins are the only data wrangling operation in canvases that expect two inputs, and that came with some unique design challenges. For example, we needed to decide if the precise connection port implied which input table would be the left and right table in the join. In other words, should connecting an upstream table to the top port of the Join node make it the left table? We landed on adding two ambiguous input ports to the Join node. A user connects two tables to the node without needing to specify at that point which is left or right. Within the join node a user can more easily see controls showing which is which, with a button to swap the two. The result is a Join node that doesn’t require precise connector positioning or forethought about table order before connecting the two input tables, and that still makes it clear how tables are understood by the join: ## **Go behind the scenes with Observable Canvases** For one of the most common methods in all of data wrangling, UI-based joins are surprisingly hard to implement well. By constructing explicit queries that avoid issues with SELECT \*, tailoring joins to react to upstream changes, and designing for simplicity and clarity, we’re making joins straightforward and robust in canvases. We continue to share behind the scenes views into how we’re designing and building Observable Canvases as we add new features, optimize user experience, and improve performance. Check out recent pieces in the series: - [Learning to play safely with fire: building interpretable AI for data analysis](/content/blog/playing-safely-with-fire-building-interpretable-ai-for-data-analysis/index.html) - [Big data, fast data visualizations: a primer on chart performance in Observable Canvases](/content/blog/big-data-fast-data-visualizations/index.html) - [Designing better out-of-the-box histograms](/content/blog/designing-better-histograms/index.html) Canvases enable fast, flexible AI-driven data analysis, and put state-of-the art interactive data visualizations and dashboards at your fingertips. [Learn more](/content/platform/canvases/index.html) and request a demo today. \]\]>https://observablehq.com/blog/new-advanced-charts-in-observable-canvases https://observablehq.com/blog/new-advanced-charts-in-observable-canvasesThu, 28 Aug 2025 00:00:00 GMTWhen building out the data visualization library in [Observable Canvases](/content/platform/canvases/index.html), we started with essential chart types like bar charts, histograms, line charts, and scatter plots. We also knew we wanted to add more advanced chart types that feel out of reach for many data analysts because they are difficult and time consuming to build from scratch. Over the past several months, we’ve been designing and building out our suite of advanced charts in canvases. These chart additions make it possible for users to generate interactive visualizations like Sankey diagrams, circle packing charts, streamgraphs, and arc maps in just a few clicks. In this post, we highlight six of our newest interactive charts in Observable Canvases. Read on for a look at each new chart type, see how they are created in a canvas, and learn about useful customization options. ## **Bump charts** In canvases, bump charts expect a categorical variable for the series and a number for the value that rankings are based on. Below, a user creates a bump chart of U.S. seafood imports by country. Data: NOAA [US Trade in Fishery Products](https://www.fisheries.noaa.gov/foss/f?p=215:2:1986023721187) ## **Beeswarm charts** Beeswarm charts visualize observations for a single quantitative variable, with jitter to avoid overlapping marks. As shown in the beeswarm chart below, encodings for color and size can represent additional variables in the chart. Here, a user creates a responsive beeswarm chart to visualize the distribution of product prices, with a different color for each overall product category. Data: [Amazon e-commerce data](https://www.nature.com/articles/s41597-024-03329-6) Beeswarm charts are a useful and visually interesting alternative to other charts that show a distribution of values, like histograms or box plots. With easy options to control dot sizes, text labels, padding, and more, you can quickly customize the look of beeswarms in canvases. ## **Sankey diagrams** Sankey diagrams show flows of quantities between different stages, locations, or groups. In a Sankey diagram, flow amounts are represented by the width of a band moving from one stage to the next. In a canvas, quickly create a Sankey diagram by selecting “Source” and “Target” variables. The default band size is based on counts of rows shared between each combination of endpoints. Update the “Size” field to visualize flow amounts based on a different quantitative variable in the dataset. Below, a Sankey diagram is built in a canvas to visualize flows of fresh yellowfin tuna imported from Asian countries to the United States in 2024. The source is the country where the tuna originates; the target is the state through which the import is processed. Size is based on the volume imported in kilograms. Data: NOAA [US Trade in Fishery Products](https://www.fisheries.noaa.gov/foss/f?p=215:2:1986023721187) You can further customize the look and layout of the Sankey using built-in options to adjust padding, alignment, and the order of source categories (e.g., in alphanumeric order, or by decreasing size). ## **Circle packing charts** Circle packing charts represent hierarchical data, with circle size usually based on a quantitative value by group (like a count or sum). They can be an alternative to other visualizations showing hierarchical data or parts of a whole, such as [treemaps](/content/learn/glossary#treemap/index.html) or [pie charts](/content/learn/glossary#pie-chart/index.html). Canvases make it easy to switch between different chart types with compatible data inputs. Below, a treemap is rendered on a canvas showing the breakdown of consumer complaints by issue and response status. For a different view, the user updates the type to a circle packing chart; the variable selections are automatically ported over to the new chart type for a fast alternative display. Data: Consumer Financial Protection Bureau [Consumer Complaints Database](https://www.consumerfinance.gov/data-research/consumer-complaints/) ## **Streamgraphs** Streamgraphs show changes in counts or values over time by group, usually applying some smoothing interpolation between values to produce a fluid or wave-like appearance (hence “stream” in the name). They are an alternative to stacked temporal histograms or area charts. Create a streamgraph in a canvas by selecting a data variable for X, a numeric value for Y, and a categorical variable for color to show smoothed changes by group over time. In the example below, a streamgraph reveals an overall increase in online orders across categories around the start of the COVID-19 pandemic. Notice in particular the expanding band size for essential items in the Home and living (red), Health and personal care (yellow), and Food and grocery (purple) categories. Data: [Amazon e-commerce data](https://www.nature.com/articles/s41597-024-03329-6) Streamgraph options include a dropdown for different date intervals (day, week, month, or year) and several different interpolation methods (catmull-rom, basis, linear, or step). ## **Arc maps** Arc maps show flows or connections between locations. We’ve recently added arc maps in canvases, currently supporting global (by country) and U.S. state connections. To make an arc map in your canvas, first select valid columns with source and target names to generate the arcs. Use the Value option to update arc thickness based on a quantitative variable. Use built-in options to change the default zoom level, choose the map center with a specific latitude and longitude, and tweak the pitch to adjust the view angle. In the example below, an arc map visualizes seafood imports to the United States in 2024, with arc thickness and color dependent on import volume from each exporting country (in kilograms). Data: NOAA [US Trade in Fishery Products](https://www.fisheries.noaa.gov/foss/f?p=215:2:1986023721187) ## **We’re busy building new charts in canvases** With built-in options for interactive Sankey diagrams, streamgraphs, beeswarm charts, and more, Observable Canvases help you create and share hard-to-build-from-scratch data visualizations in just minutes. Moving forward, we plan to bring the richness and variety of charts seen in the [Observable Plot](/content/@observablehq/plot-gallery/index.html) and [D3](/content/@d3/gallery/index.html) galleries to your fingertips in canvases. As we develop more chart types, we’ll keep sharing the newest additions and options to keep you up to date. Read our recent posts to learn how we’re designing and building fast, interactive charts in canvases: - [Big data, fast data visualizations: a primer on chart performance in Observable Canvases](/content/blog/big-data-fast-data-visualizations/index.html) - [Designing better out-of-the-box histograms](/content/blog/designing-better-histograms/index.html) Want to learn more about getting your data on the fast track from database, to business insights, to interactive state-of-the-art visualizations?  [Request a demo](/content/platform/canvases/index.html) of Observable Canvases. \]\]>https://observablehq.com/blog/canvases-summer-release-2025 https://observablehq.com/blog/canvases-summer-release-2025Tue, 26 Aug 2025 09:00:00 GMTOur biggest release yet for Observable Canvases is here. [Observable Canvases](/content/platform/canvases/index.html) helps teams make their data observable so they can turn that data into insights and action faster. We do this by reimagining the data workflow and bringing queries, tables, charts, freehand sketches, highlights, notes, and more into a computational whiteboard for data. In this release, we now offer dozens of charts that you can ship in just a few clicks. Plus, new ways to share those visualizations through dashboards and embeds. And, don’t miss the workspace management features that make managing your team in Observable even easier. Here’s everything new in Observable Canvases. ## Canvases now offer more chart types out-of-the-box Create advanced charts, like Sankey diagrams, circle packing charts, arc maps, and more, in seconds. With support for dozens of charts, canvases give you the flexibility to choose between basic charts, like [histograms](/content/blog/designing-better-histograms/index.html) and bar charts, and more advanced visualizations — without any code required. We’ve also added a new chart browser, to make it easier to select from the available options. Canvases help you create charts faster so you can generate insights faster. With the ability to quickly test different displays of data, you might recognize a trend or discover an anomaly that you otherwise would have missed. Because canvases lower the cost of chart creation, it empowers you to use visualizations to think with data. At Observable we’ve always believed that code is a prerequisite for truly custom, stunning visualizations. But not everyone has the expertise or time to build bespoke data visualizations. Plus, code-based workflows aren’t practical for the many ad hoc requests, last minute pings, and small data side quests that data analysts or data teams face. [D3](https://d3js.org/), for instance, is extremely powerful, but also complex, making it a big lift to learn and therefore inaccessible to many data practitioners. AI can reduce the effort required to create custom data visualizations with code, but it’s still a potentially time-consuming endeavor that should be balanced with the impact of the visualization. Legacy self-serve analytics tools, on the other hand, provide too many constraints that make it difficult to generate meaningful, aesthetically pleasing charts displaying the right data. With our expanding chart options, we're building a better way: a flexible, all-in-one place to build visualizations that doesn't make you trade abstraction for quality. We’re working to unify our offerings so you can choose the right tool for the job at hand — whether that’s a custom visualization created in an Observable Notebook or an off-the-shelf chart built in the canvas. ## **Canvases now support more sharing options** _If a visualization is created but no stakeholder sees it, does the chart make an impact?_ Canvases shine as an exploratory data analysis tool; a key step when communicating with data is sharing the insights that are discovered along the way. Data collaboration is powerful for getting the whole team on the same page and reducing back-and-forths, but it isn’t always appropriate or helpful to bring stakeholders into a messy canvas. This is why we’ve built new ways to share more polished presentations of data with stakeholders and decision makers. ### Dashboards We want to transform how businesses make their data visible and consumable. Enter the beleaguered dashboard. There are lots of problems with the BI industry’s current approach to dashboarding, as evidenced by the [dashboards rotting](/content/blog/five-strategies-fight-dashboard-rot/index.html) away in most BI tools. Dashboard rot is a symptom of brittle data workflows that make it hard for data teams to keep dashboards up-to-date, and that exclude stakeholders from feeling ownership and investment in the final product. Better dashboards are born from streamlined workflows and end-to-end collaborative data practices, including [visual data analysis](/content/blog/visual-data-analysis/index.html), that bring stakeholders into the fold. Data is best explored together, with shared context and conversations needed to uncover deeper insights and ensure that dashboards are useful and clear for the intended audience. Our approach to dashboards is grounded in the canvas, whereby dashboards are a natural extension of your analysis rather than an additional step and subsequent workflow. Within the data workflow, the emphasis should be on analysis — getting answers, finding value, discovering insights — with less work required to build a polished output in an entirely separate tool. Instead of selecting charts to export to another view, you simply draw a frame around the charts and tables you want to display and create a presentation link. This polished view is hosted at a separate URL without the distraction of the tool bar or nearby analysis paths on the canvas so stakeholders can stay focused on what matters most. Dashboards maintain the interactivity of the original chart, allowing viewers to quickly explore different slices of the data with built-in filtering. ### **Embedded analytics** With embedded analytics, you can get your insights from canvases to decision makers without requiring them to bookmark a new URL. Embed individual charts, or a group of charts, into your internal applications that support HTML iframe embedding. Embeds maintain the interactivity of the original charts, so embed viewers can brush and interact with existing nodes. Canvases give you the tools you need to ensure that the data displayed in your embed is up-to-date. When an embedded node is updated in the canvas, it also updates in your app. ## **New data sources** In addition to Snowflake and file upload using DuckDB-Wasm, we now support [Postgres](https://www.postgresql.org/) and [Databricks](https://www.databricks.com/). We will continue to expand the data sources we support, so if you’re interested in canvases and use a different data source, [please let us know](mailto:sales@observablehq.com). ## **Workspace management features** ### **Viewer and editor seats** We’re building a permissions system with viewer and editor roles to support both data teams and business users as they pursue collaborative analytics together. Set by owners at the workspace level, canvas editors can modify data in the canvas, share presentation links, interact with AI, and lock/unlock nodes and shapes. Viewers cannot make changes to the canvas, which means they cannot change data, move nodes, add text, or use AI. They can share presentation links with others who have access. You can lock one node, many nodes, or even the entire canvas to ensure no changes are made to your work. ### **Version control** Users need to be able to see what changes have happened to a canvas and any associated artifacts. This must include a history of changes and the ability to revert to earlier versions. With version control, you can now see and restore prior versions of a canvas to revisit past analysis paths and explorations. ## **Conclusion** Observable Canvases are now available to pilot. To stay up-to-date on all our new releases, [sign up for our newsletter list](/content/platform/canvases?modal=newsletter/index.html). Even better — give canvases a try for yourself by [reaching out to our sales team here](/content/platform/canvases?modal=demo/index.html). \]\]>https://observablehq.com/blog/essential-charts-code-snippet-observable-plot https://observablehq.com/blog/essential-charts-code-snippet-observable-plotThu, 21 Aug 2025 11:00:00 GMTA handful of essential chart types can carry you surprisingly far when analyzing, exploring, and communicating with data. This is why there are some common chart types like bar charts and scatterplots are an indispensable part of every analyst’s toolbox. In this post, we outline five essential charts that you can build with just a few lines of [Observable Plot](/content/plot/index.html) code. Observable Plot, our open-source JavaScript library, is designed for exploratory data analysis. We also share some snippets of Plot code to help get you started with visualizing data. All of these snippets are modular, so you can easily open a new Observable Notebook to start exploring and visualizing data — simply choose the chart type in a notebook and the snippet will be added automatically. So whether you’re just learning Observable Plot, or need to refresh your memory, this guide can help you quickly understand how to build some essential data visualizations. ## Core concepts in Observable Plot Here’s a quick overview of how Plot code is structured: - **Plot.plot()** initializes the plot library and renders a Plot chart. - The **mark** specifies the geometric shape used to represent the data. In the below example, dots are added to represent data in a scatterplot. - Plot favors data in **tidy format**, structured as an array of objects. You can also use an Apache Arrow table as the input data. Keep in mind, you can also organize Plot code in mark.plot() order, which you’ll see in some of our Plot documentation and examples. In this alternate style, the equivalent of the code above looks like this: ``` Plot.dot(cars, {x: "weight", y: "mpg"}).plot() ``` ## How to build a bar chart When most people think of charts, they often think of the bar chart. [Bar charts](/content/learn/glossary#bar-chart/index.html) are a reliable and dependable tool for visualizing and comparing values by category. They are among the most common charts used by data analysts in dashboards and reporting. In Observable Plot, you can use either barY for a vertical bar chart, or barX for a horizontal bar chart. You can specify the sorting you’re looking for with the [sort option](/content/plot/features/scales#sort-mark-option/index.html). Check out this example of Observable Plot code, which displays the frequency of letters used in English, sorted by descending frequency: ``` Plot.plot({ marks: [\ Plot.barY(alphabet, { x: "letter", y: "frequency", sort: { x: "-y" } })\ ] }) ``` _Bar chart showing the distribution of letter frequencies in an English-language corpus._ ## How to build a scatterplot Scatterplots are a workhorse for showing the relationship between two quantitative fields. In Observable Plot, you can use the [dot mark](/content/plot/marks/dot#dot-mark/index.html) to quickly build a scatterplot, visualizing observations across both x and y axes. The dot mark supports a vast array of options, including for custom dot size, symbol, opacity, and color blend mode. Check out the below example of a scatterplot built with Observable Plot, which visualizes the relationship between car gas mileage and horsepower: ``` Plot.plot({ marks: [Plot.dot(cars, { x: "power (hp)", y: "economy (mpg)" })] }) ``` ## How to build a line chart Another common chart used in data analysis is the line chart, which visualizes quantitative or temporal observations by interpolating between adjacent data points. Many analysts rely on line charts to show trends over time, such as for stock prices or temperature readings. In Observable Plot, you can use the [line mark](/content/plot/marks/line#line-mark/index.html) to build a line chart — check out the code snippet below, visualizing the change of Apple’s stock price over time. And explore the many [options to customize lines in Plot](/content/plot/marks/line#line-options/index.html), including [twenty different built-in interpolation methods](/content/plot/features/curves/index.html)! ``` Plot.plot({ grid: true, marks: [Plot.lineY(aapl, { x: "Date", y: "Close" })] }) ``` A line chart visualizing the change of Apple’s stock price over time. ## **How to build a histogram** Histograms are great for grouping quantitative values, and visualizing how often observations fall into those intervals, which are known as bins. Binning is the process of grouping values across a continuous range into discrete intervals. Histograms are frequently used in data explorations as they allow you to quickly see the distribution of a continuous dataset and get an immediate sense of important statistics like central tendency, spread, skew, outliers, and multimodality. By pairing [bin transforms](/content/plot/transforms/bin#bin-transform/index.html) with the [rect mark](/content/plot/marks/rect#rect-mark/index.html), you can easily build histograms with Observable Plot. The below example code generates a histogram that visualizes observations of penguins by body mass: ``` Plot.plot({ marks: [Plot.rectY(penguins, Plot.binX({ y: "count" }, { x: "body_mass_g" }))] }) ``` This might look a bit tricky at first, since it combines the rect mark with a bin transform. Here’s a breakdown of what’s happening in each part of the code above: Here is the histogram generated by the code above: ## **How to build a heatmap** Heatmaps are very effective at highlighting differences in values across a grid or map. They’re commonly deployed in data explorations to quickly spot patterns across two dimensions, or when observations are too dense to visualize effectively with a scatterplot. Similar to histograms, building heatmaps in Observable Plot uses the bin transform and rect mark. In the code snippet below, you can see an example of a heatmap that visualizes the relationship between the height and weight of Olympic athletes. ``` Plot.plot({ color: {scheme: "YlGnBu"}, marks: [\ Plot.rectY(\ olympians,\ Plot.bin({ fill: "count" }, { x: "weight", y: "height" })\ )\ ] }) ``` ## **Start building with Observable Plot** Armed with just a few handy chart types, including bars, dot, line, histogram, and heatmap, you can start exploring and analyzing your raw data. By utilizing additional features in Observable Plot, such as [transforms](/content/plot/features/transforms/index.html), [facets](/content/plot/features/facets/index.html), and [scales](/content/plot/features/scales/index.html), you’ll be able to build a vast array of data visualizations, helping you turn questions about your data into clear, insightful visualizations. Ready to roll up your sleeves and get started on your journey to develop data visualizations with Observable Plot? By copying the above snippets of [Observable Plot](/content/@observablehq/plot-gallery/index.html) code, you can dive right into your next data exploration and start making sense of your data. You can also browse the Observable Plot gallery to get inspired, and find examples that you can fork and customize so you can hit the ground running. Learn more about developing visualizations with Observable Plot with these resources: - Get answers to all your technical questions with the Observable Plot [documentation](/content/plot/getting-started/index.html) - Watch a four-part course on Observable Plot on our [YouTube channel](https://www.youtube.com/playlist?list=PLOHIJAFwtkEevglMLsR9wBnXpPK3QY_Go) - Discover [how to reshape datasets](/content/blog/reshaping-data-plot-d3/index.html) to build visualizations in Plot and D3 - Check out some [expressive and underutilized chart types](/content/blog/underused-options-observable-plot/index.html) you can create with Observable Plot \]\]>https://observablehq.com/blog/what-is-a-grid-cartogram https://observablehq.com/blog/what-is-a-grid-cartogramThu, 14 Aug 2025 00:00:00 GMTWhat if you could take a map, ignore all the messy borders, and turn it into a clean, tile-based layout — without losing its geographic meaning? That’s the idea behind grid cartograms, an engaging and powerful way to visualize spatial data. Instead of using traditional map shapes, grid cartograms represent regions like states or countries with uniform geometric tiles, arranged in a roughly geographic layout. This design makes it easy to spot patterns, compare regions side by side, and gives small areas equal visual weight next to larger ones. It’s a favorite among data journalists, policy analysts, and visualization developers alike. In this post, we’ll explore what grid cartograms are, why they’re useful (and when they’re not), how they compare to other types of cartograms, and how you can start building them in Observable. ## **What is a grid cartogram?** A grid cartogram is a type of map that represents subregions using tiles of the same shape, arranged to reflect their general geographic locations while avoiding overlap. The most common tile shapes are squares and hexagons. For example, the grid cartogram below by Joe Davies of Eurostat visualizes GDP for European countries using hexagonal tiles. [Visit the Observable Notebook](/content/@eurostat-ws/grid-cartogram/index.html) to see the code, and to toggle between hexagonal and square tiles. In the grid cartogram above, each country is represented by a single tile, with color used to indicate GDP. This gives smaller countries like Liechtenstein (LI) and Cyprus (CY) — which would be nearly invisible if scaled to their true land area — equal visual representation alongside larger European countries. An alternate approach to grid cartograms sticks to the grid layout, but each tile represents a unit of measurement (for example, one tile for every 1000 people) instead of the “one tile per region” approach shown above. The non-contiguous grid cartogram below from the [New York Times’ Learning Network](https://www.nytimes.com/2020/11/19/learning/whats-going-on-in-this-graph-2020-presidential-election-maps.html) shows outcomes for the 2020 U.S. presidential election, with each square representing a single electoral college vote: While the grid layout may seem like a constraint, you actually have quite a bit of flexibility regarding tile shapes and content. The grid cartogram below, by Observable Software Engineer Toph Tucker, uses a [unique triangular tile subdivided to show election outcomes by state for the 2020 U.S. election](/content/@tophtucker/election-2020-vote-margin-cartogram/index.html). A line extending from the top point of each triangle veers left (Biden) or right (Trump) as votes are returned, with the final vote margin intersecting the triangle base. Each tile in a grid cartogram can even house an entire additional chart to display small multiples in a meaningful spatial layout. For example, the [USGS Vizlab](https://labs.waterdata.usgs.gov/visualizations/index.html#/) team creates stunning grid cartograms to visualize streamflow by state, relative to historic records: ## **What are some of the pros and cons of grid cartograms?** Grid cartograms might be a good option for your thematic maps for a number of reasons: - They simplify complex spatial data into a clear, digestible grid that helps a viewer focus on comparing values between regions, rather than on extraneous details of state or country boundaries. - Each region is the same size and shape, it puts smaller regions that can be overlooked or underrepresented on a more even playing field alongside larger ones. - Grid cartograms can add a meaningful spatial dimension to faceted charts or small multiples, as seen in the USGS stream gauge chart above. The overarching drawback of grid cartograms is geographical inaccuracy. Because grid cartograms represent regions using simplified geometric tiles, information about the true (and relative) size and shape of each region is sacrificed. And, because tiles are snapped to a grid, their relative locations are imperfect: regions that share a border in reality can appear at a distance from each other in a grid cartogram, and vice versa. When recognizable boundaries and exact locations are eschewed for a simpler display of gridded tiles, it can be difficult for viewers to understand what they’re looking at and quickly recognize regions of interest. ## **What are some common alternatives to grid cartograms?** Depending on what you want to emphasize in your visualization, you might choose other cartogram variations. Here are some common alternatives: ### **Dorling cartogram** A Dorling cartogram, like a grid cartogram, represents all regions using the same simplified geometric shape. They differ from a grid cartogram because tiles are scaled based on the value of interest. The Dorling cartogram below, created by Observable Ambassador [Nicolas Lambert](/content/@neocartocnrs/index.html), represents each country as a circle, with circle surface areas proportional to each country’s population in 2019. Note that in a Dorling cartogram, tiles aren’t aligned to a grid. Here, they are first placed at each country’s centroid, then nudged to avoid occlusion. Visit [Nicolas’ notebook](/content/@neocartocnrs/dorling-cartogram/index.html) to explore the code. ### **Contiguous cartogram** If you just see “cartogram” on its own, that often means a contiguous cartogram, sometimes called a value-by-area map. In contiguous cartograms, regions are warped such that contiguity between adjacent regions is maintained, but the area of each reflects a quantitative value (like population, GDP, or carbon dioxide emissions) rather than the land area. Below is a contiguous cartogram, made by Nicolas Lambert and Observable Software Engineer Fil Rivière, showing GDP by country. Explore [their notebook](/content/@ambassadors/lets-make-a-dougenik-cartogram/index.html) to see the code, toggle between quantitative variables, and to see how the map changes with different levels of exaggeration. ### **Non-contiguous cartogram** One issue with contiguous cartograms like the one shown above is that, in order to maintain contiguity, regional shapes can get so warped that they become unrecognizable. In some cases, it might be worth it to trade contiguity for more accurate shapes. Non-contiguous cartograms avoid the distortion issue because they don’t try to keep adjacent regions connected, so you can show regions that are geographically accurate, while still representing a value using size. For example, the cartogram below by Karim Douieb, data scientist and co-founder of [Jetpack.ai](http://jetpack.ai/) visualizes results of the 2022 U.S. House election, with size scaled proportionally based on the number of districts in each state. See Karim’s notebook for a [thoughtful explanation of this approach](/content/@karimdouieb/us-house-election-2022/index.html), and to toggle between a traditional (contiguous) map and the non-contiguous cartogram. You can also learn more about the process of building and iterating on this spatial data visualization by watching Karim’s talk during our annual virtual event, Observable Insight, on [Rethinking Election Maps](https://www.youtube.com/watch?v=1dxP8DK9jHI). ## **Start building grid cartograms with Observable** If you’re interested in creating grid cartograms for your own spatial data visualization, here are some great resources to explore as you get started: - Observable Ambassador [Sylvain Lesage](/content/@severo/index.html) has created an excellent overview notebook describing [grid cartograms](/content/@severo/grid-cartograms/index.html) with a plethora of examples, useful resources, and considerations - See Observable Ambassador Nicolas Lambert’s fantastic collection of cartogram examples, with reusable code to make the following in [D3](https://d3js.org/): - [Rectangle cartogram](/content/@neocartocnrs/rectangle-cartogram/index.html) - [Dot cartogram](/content/@neocartocnrs/dots/index.html) - [Demers cartogram](/content/@neocartocnrs/dorling-vs-demers/index.html) - Value-by-area map (or [Dougenik cartogram](/content/@ambassadors/lets-make-a-dougenik-cartogram/index.html)) - Explore and reuse the [non-contiguous cartogram template from the D3 graph gallery](/content/@d3/non-contiguous-cartogram/index.html) \]\]>https://observablehq.com/blog/fostering-company-culture-ai-adoption-workflow https://observablehq.com/blog/fostering-company-culture-ai-adoption-workflowTue, 05 Aug 2025 00:00:00 GMTAI is advancing fast — and most of us are learning as we go. For many, AI in the workplace still feels like a black box: exciting, a little mysterious, and constantly evolving. There’s no universal playbook, and everyone is experimenting in their own way. At Observable, we believe the human side of adopting AI workflows matters most. Technology alone doesn’t create impact — people do. This is why we favor a human-centric approach to AI and why, as the People Ops leader at Observable, I’m focused on creating a company culture where curiosity thrives, experimentation feels safe, and AI naturally integrates into our workflows instead of disrupting it. This approach isn’t about chasing the latest tool or mandating its use. It’s about giving people the space to explore, learn, and grow together. Here’s how we’re making that happen as a company: ## Starting with curiosity, not control to encourage AI adoption As a data company, we decided that we needed more data points to inform a potential AI strategy. So, we started with a question: What is the role of AI in people’s workflows at Observable today? To find out, we launched a survey asking employees which AI tools they were using, how often, and for what purpose. When we analyzed the data,the answers showed that experimentation was already happening. We learned that some people were using AI daily; others experimented as needed. The key insight? AI adoption was already happening organically. Our job isn’t to push it but to support it. We also gained deeper insights into which tools teams are using, including ChatGPT, Claude, Google Gemini, GitHub Copilot, Zed, Cursor, Granola, Figma AI, and NotebookLM to: - Draft and revise messaging - Generate spreadsheet functions and debug code - Summarize meeting notes and synthesize customer feedback - Scaffold new features and even create podcast-style summaries for deeper engagement This survey is now part of our ongoing learning process. We’ll continue running it quarterly so we can track what’s changing, spot new opportunities, and keep adapting as both our needs and the technology evolve. ## Removing barriers with the AI stipend From the survey results, we also realized that people were eager to experiment but faced small hurdles, such as limited access to tools or uncertainty about which ones to try. To remove that friction, we introduced a $50 per month AI stipend, available with manager approval. It’s intentionally simple: employees can choose tools that make sense for their work without worrying about budget constraints. This stipend had a positive side-effect as it sparked organic sharing of recommendations on AI workflows and automation. Team members started threads comparing notes on what was worth trying, which tools were approved for broader use, and what made their work easier. These conversations have been invaluable for identifying high-impact tools and building a company culture of peer learning. The AI stipend is a relatively small investment for our company, but the payoff has been big: - People feel empowered to explore AI tools without hesitation - Teams are learning from one another and sharing real results - Experimentation is translating into tangible skill-building and confidence ## Embedding AI into everyday workflows Exploration is important, but AI’s real value lies in its natural integration into day-to-day work. When getting started with AI, it can be intimidating to know where to start. At Observable, we’ve observed that even minor changes can have a significant impact on our workflows. For instance, we began using AI to summarize meeting notes. It’s a small shift, but it’s reduced the burden of manual follow-ups, clarified next steps, and helped cross-functional teams stay aligned. These lightweight integrations show what’s possible: AI isn’t replacing human connection — it’s supporting it by automating a repetitive, administrative burden. This allows everyone in the meeting to be fully engaged and gives us back time. We’re intentionally taking our time before introducing policies or standardized tools. For now, we’re focused on observing what’s working in practice, learning from it, and using those insights to shape future guidance. ## Keeping AI adoption human-centered With all the buzz around AI, it’s tempting to focus only on speed and efficiency. But at Observable, we’re committed to keeping people at the center. That means giving employees room to figure things out, welcoming feedback and even “I’m still learning” moments, and encouraging open sharing of both successes and missteps. We know adoption will happen at different speeds, and that’s perfectly okay. We’ve also been clear: AI isn’t mandatory. The tools aren’t perfect, and not every use case is obvious right away. What matters most is that people feel safe to explore on their own terms. ## What’s next: Continuing to embrace AI tools We’re continuing to learn alongside our team, and our next steps include: - Running our quarterly AI survey to track adoption and sentiment - Hosting “lunch and learns” to share practical use cases - Identifying more areas where AI can reduce friction or free up time - Developing a flexible set of AI principles - Encouraging employees to attend AI events and bringing back what they learn Our vision is for AI to be a natural part of how we work, not something separate or intimidating. ## Final thoughts: Cultivating conditions to enable organic AI adoption Building an AI-driven company culture isn’t about adopting the shiniest tool or rushing to keep up with trends. It’s about creating the conditions where people feel empowered to explore, share, and learn together. HR plays an essential role in making that possible. By lowering barriers, fostering psychological safety, and keeping our focus on people, we are working to ensure that AI is not just powerful but truly human-centered. We’re still learning, and we’d love to hear from you: How is your organization approaching AI adoption? What’s working for your teams? \]\]>https://observablehq.com/blog/previewing-notebooks-2 https://observablehq.com/blog/previewing-notebooks-2Tue, 29 Jul 2025 11:00:00 GMTWe’re excited to share a preview of the next generation of Observable Notebooks! Today’s technology preview release includes: - **Observable Notebook Kit** \- An open file format for notebooks, and accompanying open-source tooling for generating static sites - **Observable Desktop** \- A macOS desktop application for editing notebooks as local files, and with a radical new approach to AI You can learn more and download Observable Desktop by visiting the [Notebooks 2.0 website](/content/notebooks/2/index.html). \]\]>https://observablehq.com/blog/three-ways-to-use-ai-for-exploratory-data-analysis https://observablehq.com/blog/three-ways-to-use-ai-for-exploratory-data-analysisThu, 24 Jul 2025 00:00:00 GMTAI promises to revolutionize data analysis, from automating data pipelines to finding meaningful patterns in massive, unstructured datasets. Businesses and data teams are rapidly adopting AI, realizing its potential to streamline their entire business analytics pipeline. But what specific tasks are a good starting point for data analysts looking to incorporate AI into their data workflow? In this post, we highlight how AI can improve a critical phase in the data workflow: exploratory data analysis. Learn how AI streamlines [data exploration](/content/learn/what-is-data-exploration/index.html) through quick data profiling, automated data wrangling with natural language queries, and by drafting exploratory charts. ## **Use AI for fast data profiling** [Data profiling](https://en.wikipedia.org/wiki/Data_profiling) is the process of gaining high-level familiarity with the structure, content, and quality of data in a database or data warehouse. In other words, it helps give a quick early answer to the question: “What’s in the data, and what can I do with it?” It is an important step in data exploration because it helps data analysts understand the data and uncover data quality issues, so they can do any necessary cleaning and decide on appropriate analysis methods. A large part of data profiling is finding summary statistics. These often include measures of central tendency (like mean, median, and mode), data spread (variance or standard deviation), and extrema (minimum and maximum values) for individual variables. Data profiling can also involve investigations of [missingness](/content/learn/glossary#missingness/index.html), duplication, and other indicators of data quality. Table-level information like table dimensions and variable types might also be explored as part of data profiling. As a one-off calculation the statistics listed above are trivial, but they can be time-consuming to find manually when working with a large database and many fields. AI allows analysts to automate data profiling to get a surface-level look at their data in a fraction of the time. Here are three example AI prompts that can accelerate initial data profiling: - _Summarize this data including table dimensions, column names, data types, and summary statistics for each variable._ - _What number and proportion of values are missing for each column?_ - _Are there any duplicate rows?_ With a better big picture understanding of the data, you're less likely to prematurely dive into analyses that are inaccurate, misinformed, or unused. This can save you from costly mistakes and wasted effort down the line. ## **Use AI to automate data wrangling** No matter how careful your data collection and quality assurance processes, [data wrangling](/content/learn/what-is-data-wrangling/index.html) isn’t going anywhere. There will always be a need to clean and reshape data to get it into a better format for downstream data visualizations and analyses. AI helps analysts speed through time-consuming data wrangling by translating natural language prompts into a series of data manipulations. A tool might do this using UI options that filter, sort, select, or derive values in sequence to complete the task. Or, the prompt might generate a SQL query that returns data in a more usable format. The latter is often called text-to-SQL or natural language to SQL (“NL2SQL” for short). Here are example AI prompts to draft new data wrangling sequences: - _Show me all orders from the Electronics & Media overall category shipped to California in 2022._ - _How many clothing orders were placed monthly from 2021 to 2023?_ - _Find the top 10 food products shipped to Florida, by total revenue._ Keep in mind that AI hallucinates, misinterprets requests, and makes mistakes — so always [remember to inspect and verify each data wrangling step](/content/blog/4-ways-to-avoid-common-gotchas-in-data-analysis/index.html). ## **Use AI to draft exploratory data visualizations** [Data visualization](/content/learn/what-is-data-visualization/index.html) is a critical part of data exploration because it helps analysts to uncover patterns, anomalies, and relationships between variables that can highlight new questions and inform subsequent analysis. But creating visualizations can be tiresome and frustrating, especially when it involves big data and tedious pre-processing to get it in a compatible shape with your chart type. It can also be intimidating. When you’ve got a bunch of tables, a blank slate starting point, and any number of chart types to choose from, sometimes it’s hard to know where to begin. AI can automatically generate charts, helping data analysts get off-and-running with quick data displays that avoid a blank slate. Here are three example prompts to generate exploratory data visualizations: - _Draft a line chart of monthly revenue over time, with a different line color for each customer income levels_ - _Create a histogram of product prices for all items in the Electronics & Media category_ - _Make a stacked bar chart of top products by quantity purchased, with fill color based on customer education level_ Ideally, AI "shows its work," exposing any code used to generate outputs and allowing you to make manual chart edits after-the-fact. ## Learn more AI has the potential to revolutionize how we work with data. From initial data profiling to automated data wrangling and drafting new charts, AI can help analysts quickly build a deep understanding of their data before diving into more formal analyses. Learn more best practices for exploratory data analysis in our recent posts: - [4 ways to avoid common gotchas in data analysis](/content/blog/4-ways-to-avoid-common-gotchas-in-data-analysis/index.html) - [End-to-end visual data analysis](/content/blog/visual-data-analysis/index.html) - [Choosing the right chart for your data](/content/blog/right-chart-for-data/index.html) \]\]>https://observablehq.com/blog/the-failure-of-self-serve-analytics https://observablehq.com/blog/the-failure-of-self-serve-analyticsWed, 23 Jul 2025 08:00:00 GMTThe promise of self-serve analytics is compelling: non-technical business users — with their deep domain expertise — could independently access, analyze, and report on data without waiting for IT or data analysts. This democratization promised faster insights, quicker decisions, and a stronger emphasis on a data-driven strategy. Many viewed self-serve analytics as a significant competitive advantage, prompting organizations to allocate substantial budgets toward tools, infrastructure, and the resources needed to deliver on this vision. Yet, despite significant investment, the reality has frequently fallen short. While self-serve data analytics has often failed to deliver on the vision, that doesn’t mean that stakeholders need to be exclusively dependent on data teams for their insights. In this post, I explore the gaps left by self-serve analytics and walk through a real-life example of how [collaborative analytics](/content/learn/glossary#collaborative-analytics/index.html) helps teams discover insights together. ## Why self-serve analytics initiatives fail Recently, I spoke with a software development company that had invested heavily in a traditional business intelligence (BI) tool. Their data team spent weeks building a comprehensive self-service interface and rolled out the tool at end-user training sessions. Enthusiasm ran high as trainers demonstrated, “Look how easy it is — just drag and drop columns to create custom bar charts instantly. You have over 100 columns to choose from!” However, only a few months later, adoption was dismal: only five out of a hundred intended users were actively utilizing the tool. Feedback revealed that users found the platform overwhelming, overly generalized, and ultimately inflexible when answering specific domain-related questions. This example is indicative of what I hear from many companies — whether a data team creates a set of pre-built dashboards that potentially go unused (contributing to [dashboard rot](/content/blog/five-strategies-fight-dashboard-rot/index.html)) or stakeholders build their own. A significant factor contributing to the failure of self-service analytics initiatives arises when the data team configures the analytics tool, provides basic training, and then instructs users to “go self-serve.” In reality, stakeholders and non-technical users often struggle to achieve their goals independently. They become confused about which data variables to use, perceive the platform as overly complex, or quickly get stuck on specific questions. As a result, they either abandon the tool or overwhelm the data team with constant support requests. The data team ends up functioning more like a helpdesk, bogged down by a backlog of unclear and often time-consuming queries. This dynamic leads to frustration on both sides, ultimately harming the return on investment. ## Common data challenges businesses face There are underlying challenges that limit the adoption of both self-serve analytics solutions and other data tools. For instance, when businesses don’t have a good [data culture](/content/blog/how-to-build-data-culture/index.html), there can be discrepancies between how key business metrics are defined and measured across the organization. Clearly defined terms such as revenue, gross margin, contribution margin, product category, and region are indispensable, as is a comprehensive semantic layer. Even if metrics are clearly defined, it can be difficult to find the correct data to query and display. When working within the confines of a self-serve BI system, stakeholders are limited to a strict set of data and might not even know what other data is being collected and stored in the data warehouse. Additionally, tool fragmentation and restrictive data governance practices impact broader analytical effectiveness, not just self-service analytics alone. ## A new approach to empowering stakeholders to explore their data The alternative to self-serve analytics is [collaborative analytics](/content/blog/collaborative-analytics/index.html), which is the practice of working with team members and stakeholders throughout the analysis process from ideation to final data product. As Observable’s co-founder and CEO Mike Bostock [put it](/content/blog/introducing-canvases-early-access/index.html), “Isolating roles is counterproductive; we should instead bring people together more efficiently.” Moreover, embedding a data analyst within each business unit significantly enhances analytics effectiveness. While the debate between centralized versus decentralized structures or direct versus dotted-line reporting remains valid, assigning dedicated data analysis resources within business units ensures deeper understanding and alignment with key business strategies. These analysts should actively participate in weekly meetings, contributing directly to strategic goals and success. Collaborative analytics isn’t just about adopting a new process and creating a shared Slack or Teams channel for data analysts to field questions from stakeholders. For collaborative analytics to truly be effective, it requires a novel way of working with tools that support this new approach. Next-generation software, including [Observable Canvases](/content/platform/canvases/index.html), has emerged to address the limitations of traditional BI tools. I refer to this as "next-generation" for three primary reasons: 1. **Infinite Canvas:** This approach enables users to visually explore and analyze data in an interactive whiteboard equipped with real-time collaboration features. Users can create notes, arrows, annotations, tables, charts, and summaries seamlessly. Even professionals accustomed to code-first environments appreciate the non-linear workflow, which promotes faster and more creative data exploration. 2. **Contextual AI Integration:** Unlike conventional tools that bolt-on AI components, next-generation platforms integrate AI deeply from the ground up. Properly implemented AI behaves more intuitively, breaking complex queries down into transparent, verifiable steps that can be visually audited. Whether joining datasets, deriving metrics, grouping information, or building custom data data visualizations, users can leverage AI for guidance and expert-level insights. Stuck? Just ask AI. 3. **Flexible Coding Options:** While no-code and low-code capabilities are central advantages, there are instances when users may require greater customization. Modern analytics tools now offer an open system with flexible options such as: - SQL cells for advanced data manipulation - JavaScript cells for custom visualizations - APIs to seamlessly import code from external applications or notebooks This approach ensures coding is not a prerequisite but instead serves as a powerful tool for those who choose to leverage it. Or put another way, code should not be a requirement, but rather your superpower. ## How collaborative analytics works at Observable To further explain, let’s use a real-world example — me. I’m Observable’s head of sales with decades of experience building go-to-market teams. Tracking and understanding our sales metrics is an essential part of the job, and I’ve learned over the years that the larger you grow, the more you depend on data. At other companies, I’ve had to rely on revenue operations teams to pull reports and the process proved time-consuming and painful at times. But, at Observable, we are all empowered to use our own technology within our roles. So, with no coding experience, I dove into an Observable Canvas to get an initial sense for what sales metrics (lead sources, pilots, cohort analyses, etc.) I could produce. I started my data exploration by visually browsing the fields in the summary table, which comes from [Attio](https://attio.com/), our CRM. This table shows me the basics about our lead funnel — where leads are coming from and the attributes of those leads, such as their company size and industry. I realized off the bat that we had too many lead sources that needed to be consolidated so I could have a more accurate picture of which sources are driving the most leads. To complete this task, I simply asked [AI](/content/platform/ai/index.html) — right in the canvas! — and after a natural language prompt it created the consolidated column for me. From there, creating a chart was as easy as picking a chart type and the fields to display from the drop-down menu. However, I wanted to add the “target” values to the chart, so I asked AI to build a line indicating the goal for each lead source using [Observable Plot](/content/plot/index.html). Proceeding to more advanced analysis, I wanted to evaluate how our ideal customer profile (ICP) related to these consolidated lead sources. With the user interface, I was able to quickly daisy-chain 4 charts with employee size, annual revenue, industry, and revenue type (new or upsell). Then, by using the “click to filter” capability on an employee range (see pink below) I could visualize the downstream effect. This proved very powerful. I could easily swap the lead chart for another category like industry or revenue, as shown below. The results indicated that we could expand our ICP to include different revenue segments. Next, I wanted to see how our pipeline was performing as we moved opportunities from initial meetings to pilots. This proved more challenging. Typically, CRMs provide point-in-time balance sheet statistics. For instance, accurately tracking how many pilots originated in Q1 isn't as straightforward as examining the pilot "stage," since completed pilots wouldn't be captured. To address this, we introduced an “exited pilot” date field to track pilot completions. I tried to use AI to combine both opportunities that are active pilots and those that were completed, using the “exited pilot” date, but ran into an error. I sent my data analyst a quick ping and shared a link to my canvas. Because AI in canvases is inspectable, my data analyst could easily identify that the exited pilot stage was a date field that needed to be converted into a string before I could derive the column. With this insight, I asked the AI to convert the column to a string, filter for Q1, and combine the two pilot columns, successfully generating the desired visualization. The next task was a more challenging cohort analysis of all stages. While working in the same canvas, my data analyst instructed me to select the [Sankey chart](/content/learn/glossary#sankey-diagram/index.html) and explained the drop down menu variables: Source (first meetings), Target (stage) and Size (# of deals). I shared my work with my engineer and he said, “this would have taken me a day or two to build with code, and you just did this in under 10 minutes!” ## How collaborative analytics fulfills the promise of self-serve The promise of empowering non-technical domain experts to explore and uncover deep insights has finally been realized. The infinite canvas facilitates easy data exploration, collaboration, and sharing. Contextual AI significantly enhances efficiency, far surpassing traditional BI methods. Additionally, when advanced customization is necessary, using the coding expertise of your data analysts becomes invaluable. My own success was enabled by close collaboration with the data team. This next-generation self-service analytics software makes all of this possible. If you’re interested in learning more, I’d be happy to walk you through Observable Canvases — request a demo with me [here](/content/platform/canvases?modal=demo&utm_source=blog&utm_content=self-serve-analytics-blog/index.html). \]\]>https://observablehq.com/blog/playing-safely-with-fire-building-interpretable-ai-for-data-analysis https://observablehq.com/blog/playing-safely-with-fire-building-interpretable-ai-for-data-analysisTue, 15 Jul 2025 10:00:00 GMTAI is having its Promethean moment, transforming how we work, how we learn, how we communicate, even how we think. And while AI’s power — its ability to code, to draft and summarize documents, to answer open-ended questions in seconds — is awesome to behold, AI is also imperfect. It makes mistakes. When it doesn’t “know” the answer, it often makes it up and hopes you don’t notice. In human terms, AI lies. AI’s mendacity will surely improve over time but also persist because, as the CEO of Anthropic says, “ [we do not understand how our own AI creations work.](https://www.darioamodei.com/post/the-urgency-of-interpretability)” This is not a simple bug to fix. This “quirk” of AI is especially concerning when analyzing data. If AI is unpredictable, how can you trust it to make business-critical decisions? While some may be willing to ignore the risks and use AI chatbots for (often incorrect) black box answers to analytical queries, we wanted our approach to be centered around **interpretable, verifiable** answers. AI’s unpredictability does not mean that it is useless; instead, like the fire of Prometheus, it must be wielded carefully. We cannot guarantee AI’s correctness, but we can make the derivation of AI answers easier to follow, and to make it more obvious when AI goes “off the rails.” AI tools must encourage scrutiny; they must help users to interpret, verify, and correct AI answers. AI shouldn’t simply give the answer; it must show its work. **Observable Canvases** [make data analysis visual](/content/blog/introducing-canvases-early-access#why-canvases/index.html), and that’s exactly what’s needed to see through AI’s lies and harness it safely. At each step — whether writing SQL or using visual operations to transform data — data is summarized visually so you can see the distribution along each column and browse sample rows. Pervasive visual summaries make it easier to inspect the data and ensure that the desired end chart or metric is valid by understanding how it was derived. The visual nature of canvases facilitates not only cross-functional collaboration amongst data teams and their stakeholders, but also collaboration with AI. ## **A human-centric approach to AI** The role of AI in canvases is to help users make decisions informed by data. We favor a human-in-the-loop approach where AI assists users, and is supervised by users, rather than automating decision-making. To further our human-centric approach, we developed the following principles: **AI “sees” the same canvas you do.** To make its behavior more predictable, and to ensure a consistent perspective, AI sees the same data — the same column summaries, the same sample rows — you see. And it likewise sees the same contents of the canvas, and what’s in the viewport, and the like. Everything on the canvas has a textual equivalent that can be included as context to AI. **AI uses the same tools you do.** There are no actions that can only be performed through AI; AI has no “extrajudicial authority”; anything AI can do, you can do by hand, too. **AI favors visual operations over “opaque” blobs of code.** The visual design of canvases makes analysis more easily read and manipulated, so AI should generate visual operations whenever possible. AI may still generate code, but only when built-in visual tools don’t suffice. We’re building an extensive library of chart types and data transformations so that AI can produce more scrutable responses. **AI should be easy to undo and redo.** AI will inevitably make mistakes; you may want to revise a prompt, try again, or even abandon AI for a manual approach. It should be easy to get back to earlier prompts and tweak them. AI-generated results should be easily undoable. **AI should work step-by-step.** AI should break problems down into multiple steps so that the chain of thought is easier to follow, rather than trying a “Hail Mary” to answer a complex question in a single step. It should be obvious when AI is actively working, and when it adds content to the canvas. AI should be interruptible; when it goes wrong, you can quickly stop it. **AI is lightweight and iterative.** AI should be immediately accessible, but also unobtrusive. AI should be fast to invoke, and fast to put away. It shouldn’t distract or beg for attention. It should not eat up precious screen real estate (leaving more space for charts). AI should only ask for attention when performing actions requested by the user. We see AI helping in the following ways: **AI accelerates tedious, unfamiliar, or underspecified tasks.** AI can quickly create queries and charts using short, natural language prompts. This saves time because you don’t have to first consult the relevant documentation (or a colleague if no documentation exists) for your data warehouse schema, for your data warehouse’s SQL dialect, or the like. AI can also translate vague questions (such as _how has product usage changed_?) into quantitative metrics that are a starting point for analysis. **AI teaches you how to use canvases.** AI is the ultimate in-product help because it tailors its responses to your specific needs. AI can demonstrate how to use canvas features either in isolation or in combination. And when best practices are unfamiliar or tedious — such as adding annotations to explain, summarize, or capture context — AI can both demonstrate and automate. **AI is a creative partner.** If you’re looking for inspiration, such as common industry metrics, ask AI for ideas. AI can also serve as a proactive investigator, noticing patterns or anomalies in the data and suggesting closer examination. ## Try it today We’re excited to introduce AI in Observable Canvases, and help you radically accelerate data analysis while retaining confidence and trust in the veracity of your insights. If you’d like access to canvases, please [sign up now](/content/platform/canvases?modal=canvas_beta/index.html). We’re currently enabling early access for data teams, with general availability coming soon. \]\]>https://observablehq.com/blog/reshaping-data-plot-d3 https://observablehq.com/blog/reshaping-data-plot-d3Thu, 26 Jun 2025 00:00:00 GMTBetween the [Observable Plot](/content/@observablehq/plot-gallery/index.html) and [D3 galleries](/content/@d3/gallery/index.html), and thousands of public [Observable Notebooks](/content/platform/notebooks/index.html) shared by community creators, there’s an expansive collection of stunning, interactive, and openly shared data visualizations waiting to be reused. If your data is already in the same format used in the data visualization you’re interested in, then you might be able to simply drop in your data, update some variable names, and see your own version of the chart appear. But often, that’s not the case, and you’ll need to do some [data wrangling](/content/learn/what-is-data-wrangling/index.html) to get your data into the expected format. In this post, we describe data formats expected by Observable Plot and D3. We start by explaining tidy data — the most common data format used by Plot charts — and showing how to get data from wide or nested structures to tidy format. Then, we cover several data shapes that appear frequently in D3 examples, including aggregated values and nested arrays. For each, we share examples in a [companion notebook](/content/@observablehq/reshape-data-for-plot-and-d3/index.html) with code that you can explore and reuse to transform your own data into the expected shape. ## **For Observable Plot: keep things tidy** [Observable Plot](/content/learn/glossary#observable-plot/index.html) is our open source JavaScript library for data visualization, which follows the [grammar of graphics](https://vita.had.co.nz/papers/layered-grammar.html) style to build and customize charts layer by layer. Whether you’re building from scratch in Plot or reusing existing examples, you’ll generally want your data in [tidy data](https://r4ds.had.co.nz/tidy-data.html) format. Tidy data is a specific way to organize data where each variable is a column, each observation is a row, and each cell contains a single value. Organizing data in this way helps you write more efficient code to generate visualizations, while letting Plot handle operations for things like grouping, aggregation, and faceting. That’s more efficient than layering separate marks for each series or group, for example, to create a multi-series line chart from values spread across multiple columns. And, a consistent and predictable tidy data format makes it easier to switch between data sources and chart types. Most often in Observable, you’ll see tidy data stored as an array of objects. For example, when you read in a CSV file using `FileAttachment`, you get an array of objects where each record is contained within an object, with key-value pairs indicating the field and corresponding value. In JavaScript, data is commonly read in as arrays of objects with key-value pairs indicating the variable name and value for each record as shown above for web Los Angeles city web traffic. Data: [LAcity.org website data](https://catalog.data.gov/dataset/lacity-org-website-traffic). Plot also supports columnar data formats, like [Apache Arrow](https://arrow.apache.org/overview/) and [Parquet](https://parquet.apache.org/). While these often look similar to data stored in rows, for example in a CSV file, they are actually quite different under the hood. In Arrow and Parquet formats, values for any field are stored together in columns. This, along with efficient compression, helps keep data access and transformation fast because you can retrieve just the variables you need for your visualization or analysis. In Plot, variables from columnar data can be used directly as [channels](/content/plot/features/marks#marks-have-channels/index.html) with the column name encoded as a string. In contrast, when referencing variables from row-based data, Plot uses an accessor function to return an array of objects containing the data. Thus, while row-based data are the more common data format in JavaScript, columnar data formats are more performant and actually require less transformation in Plot behind the scenes. ### **Getting data into tidy format** There are many different ways data can be uniquely untidy (as [Hadley Wickham](https://hadley.nz/) says: “Like families, tidy datasets are all alike but every messy dataset is messy in its own way”). Wrangling untidy data to get it into tidy format can be similarly bespoke. Here, we’ll highlight one common data transformation: pivoting data from wide to long format. For our example, we’ll use the sample data below called **waveHeight**, which contains mock maximum wave heights at three locations on various days. This data is untidy because wave heights are spread out over three columns instead of occupying one, and the key names (“naples,” “rincon,” and “campus\_point”) are actually values of the location variable. We want to pivot the data so that location and wave height each occupy their own column. There are a number of ways to do this JavaScript, for example using the`` [`pivotLonger` function in](https://pbeshai.github.io/tidy/docs/api/pivot/) [tidy.js](http://tidy.js/). Or, we could use the code below to transform **waveHeight** into tidy format in JavaScript: ``` const waveHeightLong = []; for (const row of waveHeight) { for (const location of ["naples", "rincon", "campus_point"]) { waveHeightLong.push({ date: row.date, location, height: row[location] }); } } ``` The output from the code above is a new array of objects where each variable (date, location, and wave height) is a single ‘column’: Explore the [live code to do the wide-to-long pivot](/content/@observablehq/reshape-data-for-plot-and-d3#cell-18/index.html) shown above. Want an even simpler option? Mike Bostock recently shared a [`longerY` function](/content/d/1913762d16628db8/index.html) that can do the transformation right within your Observable Plot code! ### **A unique case: a simple array of values** Sometimes, you may just have an array of values that you want to visualize, which is basically tidy data with a single variable. Both Plot and D3 happily accept arrays of values as a chart data source. For example, if the array `[1, 5, 10, 11, 25]`is stored as **simpleArray**, we can make a bar chart in Plot as follows: ``` Plot.plot({ marks: [\ Plot.barY(simpleArray),\ Plot.ruleY([0])\ ] }) ``` The code above produces the bar chart shown below, where index is shown on the x-axis, and corresponding values are represented by the bar height: See how to similarly create a [bar chart from an array of values in D3](/content/d/d17a7f5ad3924938#chart/index.html) and [explore the live code](/content/d/d17a7f5ad3924938#cell-37/index.html) used to create the charts above. ### **From nested data to flat tables** What if your data starts off in a nested structure, but you want to create a data visualization using Observable Plot? Plot can generate charts that represent a nested hierarchy, for example in the [tidy tree](/content/@observablehq/plot-tree-tidy/index.html) and [cluster diagrams](/content/@observablehq/plot-cluster-diagram/index.html), but still expects flat rectangular data. We can use [`flatMap`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/flatMap) in JavaScript to flatten nested information and return tabular data that still captures the hierarchy. For example, here are the first entries in a small nested dataset representing package distributions from large hub cities, to intermediate distribution centers and then to final destination towns: ``` shippingNetwork = [\ {\ hub: "Chicago",\ distributors: [\ {\ city: "St. Louis",\ towns: [\ { name: "Springfield, IL", packages: 120 },\ { name: "Columbia, MO", packages: 95 }\ ]\ },\ {\ city: "Indianapolis",\ towns: [\ { name: "Bloomington, IN", packages: 110 },\ { name: "Lafayette, IN", packages: 90 }\ ]\ }, …\ ```\ \ We want to get this nested data into a flat table with two columns: one, which we’ll call **location**, representing the distribution path from hub to city to town in a single string, separated by a delimiter (we’ll use a dash), and a second called **packages** with the package count by town.\ \ The code below transforms the data using `flatMap`:\ \ ```\ flatShippingData = shippingNetwork.flatMap(hubEntry =>\ hubEntry.distributors.flatMap(distributor =>\ distributor.towns.map(town => ({\ location: `${hubEntry.hub}-${distributor.city}-${town.name}`,\ packages: town.packages\ }))\ )\ );\ ```\ \ This produces the following table (only the first 6 lines shown):\ \ \ \ With our nested data represented in a flat table, we can use the [Plot.tree](/content/plot/marks/tree/index.html) mark, specifying the variable containing the path along with the delimiter to create a nice tree diagram:\ \ ```\ Plot.plot({\ axis: null,\ margin: 10,\ marginRight: 100,\ marks: [\ Plot.tree(flatShippingData, {path: "location", delimiter: "-"})\ ]\ })\ ```\ \ \ \ The format above works with the Plot.tree mark. But you might be thinking: _that’s flat, but not exactly tidy_. Which is true, since multiple values (hub, city, town) for a single record are actually combined into one cell (e.g. “Chicago-Milwaukee-Madison, WI”), breaking the tidy principle of “one value per cell.”\ \ If you _do_ want to make the above data textbook tidy, you can access the individual hubs, cities, and towns as separate fields, instead of pasting them together as we did above:\ \ ```\ tidyShippingData = shippingNetwork.flatMap(hubEntry =>\ hubEntry.distributors.flatMap(distributor =>\ distributor.towns.map(town => ({\ hub: hubEntry.hub,\ city: distributor.city,\ town: town.name,\ packages: town.packages\ }))\ )\ );\ ```\ \ Which returns the tidy data below:\ \ \ \ [Explore the code](/content/d/d17a7f5ad3924938#cell-167/index.html) we used to transform nested data into flat tables.\ \ ## **For D3: know the usual suspects**\ \ Plot’s concise API helps you build charts with fewer lines of code, but also places a bit more constraint on data inputs. [D3](https://d3js.org/), in contrast, gives you total flexibility in terms of both what you build, and the structure of data you use. That’s a bit of a double-edged sword, since more flexibility can lead to more variation in [D3 examples](/content/@d3/gallery/index.html), compared to the predictable tidy data format seen almost exclusively in [Plot examples](/content/@observablehq/plot-gallery/index.html).\ \ Still, there are several data structures used repeatedly as D3 inputs. Here, we highlight a few of the most common.\ \ ### **First things first: yes, tidy data still works with D3**\ \ D3, like Observable Plot, can happily ingest tidy data from flat tables. For example, the [scatterplot with shapes](/content/@d3/scatterplot-with-shapes/index.html) chart in the D3 gallery uses the [classic irises dataset](https://en.wikipedia.org/wiki/Iris_flower_data_set) in tidy format to visualize flower dimensions, with color and symbol distinguishing between three iris species. Complex visualizations like the [Marimekko](/content/@d3/marimekko-chart/index.html), [radial sunburst](/content/@d3/radial-stacked-bar-chart/2/index.html), and [parallel coordinates](/content/@d3/parallel-coordinates/index.html) charts also use tidy data.\ \ But many D3 examples _don’t_ start with tidy data. Examples might use aggregated values by group, or nested data structures to capture hierarchies. Even for the same chart type, you might see alternative data structures as inputs. For example, compare these two very similar slope graphs, [one which uses tidy data](/content/@d3/cancer-survival-rates/2/index.html) as an input, and [one that ingests data in wide format](/content/@d3/slope-chart/3?collection=@d3/gallery/index.html). A common task when using D3 examples is to analyze the shape of the data they expect, which is often different from what you currently have. Here’s how to get from tidy data to other formats commonly seen in D3 visualizations.\ \ ### **Aggregated values by group**\ \ Let’s say you’ve read in some nice tidy data, with a row for each observation. When inspecting an example chart in the D3 graph gallery, you find that the data input isn’t the original raw values; it’s been aggregated by group. For example, the [horizontal bar chart](/content/@d3/horizontal-bar-chart/2/index.html) example uses pre-aggregated letter frequencies as the data input, and the [donut chart](/content/@d3/donut-chart/2/index.html) uses total counts by age group in the United States.\ \ If you need to get from raw records to aggregated values by group, [`d3.rollup`](/content/@d3/d3-group/index.html) ``makes it straightforward to do so in JavaScript.\ \ Let’s find summary values using the [penguins dataset](https://journal.r-project.org/articles/RJ-2022-020/), which contains size measurements for 344 individual penguins of three species (Adélie, chinstrap, and gentoo). A subset of the penguins data is shown below:\ \ \ \ We can use `d3.rollup`to find the total number of penguins by species as follows:\ \ ```\ penguinCounts = d3.rollup(penguins, (v) => v.length, (d) => d.species)\ ```\ \ Which returns counts by species, as a [Map object](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Map):\ \ \ \ To get from a Map to the more commonly expected array of objects, we can use the [`Array.from`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/from) ``method as shown below:\ \ ```\ Array.from(penguinCounts, ([species, count]) => ({ species, count }))\ ```\ \ Which returns the following:\ \ \ \ And, rollups aren’t confined to counts. You can use them to find the sum, mean, maximum, or any other custom metric by group. For example, the code below returns the mean penguin body mass by island:\ \ ```\ d3.rollup(penguins, (v) => d3.mean(v, d => d.body_mass_g), (d) => d.island)\ ```\ \ Which produces:\ \ \ \ Want to aggregate values using SQL instead? Use a [GROUP BY clause](https://duckdb.org/docs/stable/sql/query_syntax/groupby.html) to find summary values by group.\ \ ### **Nested structures for hierarchical charts**\ \ Earlier in this post, we showed how to get from nested arrays to flat tables with paths representing nodes in a hierarchy, so that we could produce a simple dendrogram in Plot. Now we’ll consider the reverse: what if you _have_ data in flat, tidy tables, but the D3 example you want to use is expecting a nested array?\ \ This is quite common in the D3 gallery. For example, the [zoomable circle packing](/content/@d3/zoomable-circle-packing/index.html), [zoomable sunburst](/content/@d3/zoomable-sunburst/index.html), and [treemap](/content/@d3/treemap/2/index.html) charts each use nested data as the input.\ \ In this example, we’ll start with tidy data on dam characteristics, capacity, condition, and jurisdiction in the United States from the [National Inventory of Dams](https://nid.sec.usace.army.mil/). The first several lines of the data are previewed below, and stored as **dams**:\ \ \ \ Below, we use [`d3.groups`](https://d3js.org/d3-array/group), with some additional wrangling, to return nested data for nodes based on the following variables: Primary Purpose, Primary Dam Type, Primary Owner Assessment, Condition Assessment, and Hazard Potential Classification:\ \ ```\ hierarchicalDams = {\ function group(data, [name, ...path]) {\ return d3.groups(data, (d) => d[name])\ .map(([value, rows]) => path.length\ ? { name: value, children: group(rows, path) }\ : { name: value, value: rows.length }\ );\ }\ return {\ name: "damsHierarchy",\ children: group(dams, [\ "Primary Purpose",\ "Primary Dam Type",\ "Primary Owner Type",\ "Condition Assessment",\ "Hazard Potential Classification"\ ])\ };\ }\ ```\ \ The output of the code above is nested data, an expanded portion of which is shown below:\ \ \ \ This is in the same format used in the [zoomable icicle chart](/content/@d3/zoomable-icicle/index.html) in the D3 gallery. By simply replacing **data** in that example with **hierarchicalDams** created above, we can produce an interactive icicle chart for quick exploration of dam purpose, type, and condition:\ \ Explore the [live code used to transform data and create the chart above](/content/@observablehq/reshape-data-for-plot-and-d3#cell-197/index.html).\ \ ## **Get your data into shape**\ \ Whether you’re using Observable Plot, D3, or another data visualization library, you’ll need to do some data cleaning and transformation to get it into the expected shape. In Plot, that most often means wrangling data to get it into tidy format. D3 examples consume a wider variety of data formats including tidy tables, aggregated summaries, and nested structures representing hierarchies.\ \ Knowing some of the most common data formats and transformation methods will help you quickly reuse and build on the thousands of existing examples made with Plot and D3.\ \ Visit our Plot and D3 resources to find inspiration for your next data visualization:\ \ - [D3 gallery](/content/@d3/gallery/index.html)\ \ - [D3 documentation](https://d3js.org/)\ \ - [Observable Plot gallery](/content/@observablehq/plot-gallery/index.html)\ \ - [Observable Plot documentation](/content/plot/index.html)\ \ \ If you’re new to Observable, explore our suite of tools for fluid, collaborative [data analysis](/content/learn/what-is-data-analysis/index.html) and visualization at [observablehq.com](/content/site-root.html).\ \ \]\]>https://observablehq.com/blog/big-data-fast-data-visualizations https://observablehq.com/blog/big-data-fast-data-visualizationsWed, 11 Jun 2025 00:00:00 GMTModern companies are investing in data warehouses that centralize valuable information about their operations, products, finances, and more. Those warehouses often contain millions upon millions of records that can enable smarter, data-driven business decisions. But working with such massive datasets can quickly bog down data exploration and [data visualization](/content/learn/what-is-data-visualization/index.html) tools, leaving analysts waiting for charts to generate — if their tools don’t crash while trying. In [Observable Canvases](/content/platform/canvases/index.html), we want users to spend time answering questions with their data, not waiting for visualizations to load. Here, we describe why charts that represent a lot of data tend to be slow, and how we keep charts fast and useful in canvases. ## **Why big data visualizations are slow to load** Data visualizations that pull in huge amounts of data are slow for several reasons. In business analytics, the records you want to visualize often live in a cloud data warehouse. If you’re writing queries that return large numbers of records, you might be in for a long wait due to database latency (how long it takes for the database server to process and return a query response) and [network latency](https://developer.mozilla.org/en-US/docs/Glossary/Latency) (how long it takes to transfer data across a network, often from the database to the client). Generating a chart from a large number of points also takes time, whether [rendered server-side](/content/plot/getting-started#plot-in-react/index.html) or built on the client in JavaScript, like we do in canvases. Adding millions of marks to a chart area is computationally expensive, and might surpass your device’s processing power. Small changes to chart options, like updating the [blend mode](https://developer.mozilla.org/en-US/docs/Web/CSS/mix-blend-mode) or mark [opacity](https://developer.mozilla.org/en-US/docs/Web/CSS/opacity), can tank performance. When you reach the limits of what your computer, app, or graphical software is capable of, you can be left watching a spinning wheel that eventually turns into a crash notification. Visualizing large datasets can get bogged down while waiting for data from your company’s data warehouse, and during computationally expensive chart rendering. When data visualizations are slow to render, you’re left waiting for a new version of your chart to appear each time you make a change. Waiting for charts to load stifles experimentation, iteration, and fluid [data exploration](/content/learn/what-is-data-exploration/index.html). But it doesn’t have to be that way. Here’s how we keep charts snappy in [Observable Canvases](/content/platform/canvases/index.html). ## **How we make faster charts in canvases** In [Observable Canvases](/content/blog/introducing-canvases-early-access/index.html), we keep data visualizations fast in part by using automated queries that return a smaller, aggregated version of the data at a high enough resolution to still produce rich and responsive charts. The goals of doing so are to cut down how much data is transferred, and reduce the amount of subsequent data processing needed in your browser in order to generate a chart. Let’s consider an example. We want to make a [histogram](/content/learn/glossary#histogram/index.html) to explore the distribution of 8 million order prices from an online store. One way to make the chart is to write a query that returns all 8 million individual records and then do the necessary binning and counting client-side when the chart is created. That could be slow at both the data transfer and chart rendering bottlenecks. Instead, we send an optimized query to the database server that returns data aggregated at a level of resolution needed to make the chart but substantially smaller than the raw data. For our specific histogram example, that query might return counts of 8 million records in 1000 bins — which is far more bins than any reasonable histogram should contain. In that case, you have reduced your starting point from 8 million records to just 1000 values (one count per bin) plus information about the bin endpoints. After a first pass that returns aggregated data at a higher resolution than what your final chart needs, you can then do subsequent aggregations for coarser bins to produce the final histogram. In Observable Canvases, charts are fast because SQL queries return a smaller subset or aggregated version of the data instead of returning all individual values represented in the chart. In the example above, the SQL query returns aggregated counts, but at a much higher resolution than what’s needed for the final chart. Charts made this way can still be responsive, because they have access to the aggregated but higher resolution data returned by the original query. We don’t need to run additional SQL queries against the database to re-bin values at this point: we can simply let the data visualization tool (in this case, Observable Plot) do the work of additional aggregation. For example, in the histogram below the number of bins increases as a user expands the chart width to reveal more detail as space becomes available. When the returned data is aggregated at a high resolution, charts can still be responsive as shown in the histogram above. The responsive granularity helps keep charts clear and interpretable as dimensions change. ## **Better exploratory charts don’t always show** **_everything_** Returning an aggregated or abbreviated version of the data keeps visualizations fast, but may also leave you feeling that something is lost in the process. We get it. It can be comforting to have each data point in hand, ready to be individually inspected…even if that never actually happens. We also know that with really large datasets, speed isn’t the only issue. When charts try to show too much, they quickly become visually overwhelming and less clear. Meaningful patterns get muddled in the noise. Data aggregation, binning, and thoughtful truncation can produce more digestible charts that help viewers focus on important patterns, instead of getting lost in the weeds. That’s why, in some cases, we aggregate data in our first pass SQL query as described above, then do even further simplification for the final chart. For example, the SQL query for our built-in [scatterplot](/content/learn/glossary#scatter-plot/index.html) does some initial aggregation by counting observations within bins based on x- and y-coordinates. At most only 10,000 bins (the top 10,000 by count) are displayed; text annotation ensures that a user knows when additional bins are hidden. These conditions produce a simpler but still information-rich chart that helps viewers to discern major patterns in their data. And, viewers can lean on chart interactivity in canvases to get a closer look. Brushing over a region of the scatterplot below will produce a focused version of the chart, automatically re-binning the query with smaller extents to show the selected data at higher granularity. A default scatterplot in a canvas, visualizing outdoor temperature and electricity imports for Norwegian buildings. Dot size differs based on the count of observations in each 2-D bin that are returned by the “first pass” SQL query. Note the warning in the top right informing a viewer that values are truncated. Data: [Lien et al. (2025)](https://www.nature.com/articles/s41597-025-04708-3) Carefully aggregated and abbreviated data can be a win-win for speed and clarity, especially in the [messy middle of data analysis](/content/blog/visual-data-analysis/index.html). But you’re not constrained to using our built-in charts in canvases. You can always write your own custom queries, and build bespoke charts in JavaScript, to create different (albeit, potentially slower) views of your data. ## **Get past data visualization bottlenecks** In Observable Canvases, built-in SQL queries can return smaller, aggregated versions of data to create fast and clear charts — no matter how many records you’ve got. And, you don’t have to follow our recipe for speed: you can build your own SQL queries and charts in JavaScript for fully customized analysis and visualization. Learn how the [speed, fluidity, and flexibility of data work in canvases](/content/canvases/index.html) helps your team and stakeholders to answer and uncover business questions together. \]\]>https://observablehq.com/blog/how-to-build-data-culture https://observablehq.com/blog/how-to-build-data-cultureTue, 27 May 2025 00:00:00 GMTThese days, business leaders and employees realize the promise of data to optimize marketing campaigns, drive product adoption, and gain a competitive advantage. Many companies have made huge investments in data collection, storage, and pipelines that will bring that promise to bear. In short: no one disputes the immense value that data can bring to organizations. Data culture is the organizational mindset and practice of using data to improve operational understanding and make better business decisions. While the collective recognition that [data-driven thinking](https://online.hbs.edu/blog/post/data-driven-decision-making) is good for business is important, that on its own won’t magically result in a strong data culture. Putting data culture into practice takes effort, education, and communication to help everyone at an organization understand how data can improve outcomes of their work. Data analysts are often the people spending the most time in the weeds with company data on a day-to-day basis, making them particularly well positioned to influence how teams work with and make decisions using data. Here, we share practical ways that data analysts can help build a strong data culture within their organization by increasing the visibility of data work, including collaborators in analyses, and sharing knowledge as they learn. ## **Increase the visibility of data analyses** People are busy, often juggling multiple projects and deadlines at the same time. With so much going on, data can quickly fall out-of-sight and out-of-mind. When that happens, [dashboards begin to rot](/content/blog/five-strategies-fight-dashboard-rot/index.html), data fades from the spotlight, and teams slowly revert to less data-driven means of decision-making. Unless, of course, someone is making sure that data stays front-of-mind. Data analysts are often the right _someone_ to keep data on everyone’s radar for several reasons. Data analysts are often the ones building and sharing data analyses, so they also know where a lot of analyses live. Sounds trivial, right? But you might be surprised how often people don’t return to a useful dashboard simply because they never bookmarked the page, have forgotten how to find it, and are now too embarrassed to ask. Data analysts can often quickly point teammates to the right section of a dashboard, or page in a report, to get the answers they need. Second, since data analysts tend to keep a close eye on the data, they may be the first to realize and flag when notable changes or unexpected patterns emerge. By flagging data changes, analysts can provide both a timely heads up about a specific discovery, and a more general-purpose reminder that the data is there and ready to be used. At most organizations, however, data analysts are stretched thin, so manually sending reminders about where to find data, or how it’s changing, isn’t always realistic. Here are some efficient ways that analysts can help keep data front-of-mind, and easily findable, for their coworkers: - **Pin or bookmark relevant content.** Keep useful analyses right at their fingertips by pinning key data products to relevant internal channels - **Automate reminders and notifications.** Never underestimate the capacity for teammates to forget about the analysis you’ve built at their request. Consider setting up automated reminders or notifications to help your teammates revisit data at a cadence that works for them, or when an important threshold is reached. - **Document your data analysis catalogue.** Analyses quickly start showing up in different places like dashboards, reports, slide decks, and computational notebooks. When analyses live across locations, they can be hard to keep track of. Help your teammates help themselves by compiling information about where to find things. Start simple: a list of links in a Google doc, or added to your company handbook, might be all you need. A strong data culture requires a level of consistent immersion in company data so that when people have a question, their first thought is how they can use data to answer it. Analysts can help to ensure that useful data work isn’t overlooked in company decisions by increasing the discoverability of data analyses. ## **Practice collaborative analytics** When [data analysis](/content/learn/what-is-data-analysis/index.html) is done in a black box, coworkers outside of the data team can feel detached from the data and skeptical of the findings, which decreases the value they place on data analysis and makes them less likely to look to data first in the future. For coworkers to stay invested in data work, they need to be meaningfully involved in the analysis process. That doesn’t mean that everyone will become a data analyst. It _does_ mean that data analysts should practice [collaborative analytics](/content/blog/collaborative-analytics/index.html) to include teammates, who can share valuable experience and expertise, throughout the course of a project. Collaborative analytics improves data analyses by incorporating diverse feedback from start to finish. A good side-effect is that your collaborators build a greater sense of investment in the work, and trust in the findings, when they participate directly in data analyses. “All together now!” is a great slogan, but how can data analysts actually put collaborative analytics into practice at their organization? Here are a few ways to start: - **Use tools that let everyone participate.** When possible, use tools that let your colleagues interact with and participate directly in the analysis by interacting with charts to explore different scenarios, leaving comments, or adding annotations right where you’re working. - **Show your work.** Try to give collaborators a clear view into your analyses, for example by using [pervasive visuals](/content/blog/visual-data-analysis/index.html) that help them understand data transformations without reading code. Working transparently helps colleagues and stakeholders follow your work from database to final decision, which builds understanding and trust. Welcoming your coworkers into the data fold with collaborative tools and transparency helps to build greater value and trust in data analyses. ## **Share as you learn** A core piece of strong data culture is [data literacy](/content/learn/what-is-data-literacy/index.html). For data to be used in companywide decision-making, employees need a baseline familiarity with data so they can make sense of analyses. Data analysts typically have a strong foundation in [data wrangling](/content/learn/what-is-data-wrangling/index.html), analysis, data visualization, and interpretation. That puts them in a unique position to be a hub of knowledge growth and continued learning around data. Sharing knowledge doesn’t have to be a huge added lift. Small, frequent actions that incrementally build a team’s data literacy can make a big difference long-term. Here are a few lightweight ways a data analyst can contribute to a stronger data culture at their company: - Repost insightful articles and blog posts to team channels - Share effective data visualizations (and explain what makes them work well) - Host or facilitate a lunch-and-learn session on a relevant data topic - Hold occasional “office hours” where coworkers can bring their data-related questions There are more involved ways a data analyst can grow data literacy across their organization, like  leading workshops or giving internal talks. But when it comes to data literacy, every improvement counts. And, you never know when the thing you share will be exactly what a coworker needs to get over their next data hurdle. ## **Learn more** Data analysts have an important role to play in building a strong data culture at their organization. By bringing visibility to analysis, engaging colleagues in the work, and creating learning opportunities that improve data literacy companywide, analysts can help everyone confidently interpret and work with data for more informed business decisions. \]\]>https://observablehq.com/blog/designing-better-histograms https://observablehq.com/blog/designing-better-histogramsThu, 22 May 2025 00:00:00 GMTHistograms are essential for [data exploration](/content/learn/what-is-data-exploration/index.html). So much so, that most graphical and statistical software allow users to create them with a single click or command. Both common and useful in [business intelligence](/content/learn/what-is-business-intelligence/index.html), histograms are often the first [data visualization](/content/learn/what-is-data-visualization/index.html) an analyst creates when exploring metrics like customer lifetime value, acquisition cost, and session lengths of website visits. Histograms visualize the frequency distributions of numeric variables, with bars representing the binned counts of observations over a range of values. They provide a clear and intuitive look into the shape of the data, giving viewers a quick look at important characteristics like skew, modality (e.g. unimodal or bimodal), and outliers. Designing default histograms may sound like a straightforward task. After all, all it takes is determining a reasonable number of bins, counting up the observations in each, and plotting the counts as vertical bars…right? Wrong. Behind the scenes, a number of careful decisions go into building useful histograms that work well for a wide range of data — and there’s a surprising amount of gray area to navigate along the way. Here we describe three interesting challenges we encountered, and how we decided to handle them, while designing out-of-the-box histograms in [Observable Canvases.](/content/blog/introducing-canvases-early-access/index.html) ## **Handling outliers** In business analytics, finding a normal and [mesokurtic](https://en.wikipedia.org/wiki/Kurtosis) distribution is rare. You’re more likely to encounter skewed distributions, with most values bunched together on one end and a small number of values extending out in distant tails. For example, the histogram below shows the distribution of product prices for a random sample of 200,000 Amazon e-commerce purchases: Outliers can expand the x-axis scale far beyond a range where the bulk of observations exist, resulting in largely empty histograms with just a few visible bars. Data: [Berke et al. 2024](https://www.nature.com/articles/s41597-024-03329-6). This histogram isn’t _wrong_. But it also isn’t particularly useful — or nice to look at. The issue is that the full range of product prices shown on the x-axis (maximum: $3,579) extends far beyond the range within which most observations exist, which looks to be between about $0 and $200. As a result, the majority of values are cramped into a handful of narrow bars, while the rest of the chart area looks like wasted blank space. Given this common scenario, we wanted to design histograms that give viewers a more informative look at most of the data, while (1) delivering an honest representation of the overall distribution, and (2) without crudely chopping off the tails. We landed on the following approach in [Observable Canvases](/content/platform/canvases/index.html), which gives more space to the bulk of the data while still providing useful information about the range and number of outliers that lie beyond: Since values between the 1st and 99th percentile are always included as part of the bulk data determining the histogram domain, at most 1% of observations on each end — and often less than that — are contained within the outlier bars. We include values beyond the 1st and 99th percentiles so that observations at or close to the bounds of that range are not excluded based solely on position. For example, if the first 1,000 values are all zeroes, and the 1st percentile occurs at zero #372, it wouldn’t make sense to segregate the first 371 zeroes as outliers in a separate bin. The outlier bin differs from the others in an important way: its width, though equal in pixels to the others in the histogram, actually encompasses the entire range of outlier values. That’s why it’s crucial to make the outlier bin visually distinct (which we do using gray, hatched styling), and to provide more information about the values it contains. In a canvas, brushing over the outlier bin in the histogram above reveals both the count ( _n_ = 1,450, or 0.725% of all values) and total range of values ($200 to $3,579) it contains: Are there tradeoffs? Sure! Do we think this is the one “correct” way? No! Any time we rescale different pieces of the data in different ways, it creates room for misinterpretation. And, there are times when a viewer will want to see an expanded view of outliers. However, we think this approach strikes a nice balance between a clearer and more informative look into most of the data, while still keeping outliers in sight. ## **Making sure small counts don’t disappear** The strategy above addresses an x-axis issue, but there’s also a common y-axis issue in histogram design: the disappearance of small counts. When a bar representing a small count is plotted alongside much larger bars on a linear scale, it can become indistinguishable from a completely empty bin. For example, the histogram below shows the distribution of commercial electricity rates across different U.S. zip codes. Due to the relatively large y-axis scale, the bar of height 7 (red arrow) is indistinguishable from the bar of height 0 to its right (blue arrow). In [exploratory data analysis](/content/blog/visual-data-analysis/index.html), it’s hard to know what data to dig into further if you don’t even know where data exists. That’s why we think it’s important for viewers to see the difference between nothing and _something_ — even if that something is very small. We explored a number of approaches to uncovering these small but extant bins. One option we considered was log transforming counts to compress the y-axis scale and give more ink to smaller bins. But log transforms and scales are notoriously hard to interpret, and can leave viewers with a false sense of parity between counts that are actually quite different. And, as a kicker, there’s that nagging math issue that arises when you have bins with _n_ = 0 (since log(0) is undefined), which inspires some creative workarounds. Ultimately, we decided to stick with untransformed counts and a linear y-axis scale. But, we still thought it necessary to visually distinguish between bars with n = 0, and n > 0. We landed on setting an absolute minimum pixel height for any bars with a non-zero count. The histogram below highlights the same bins as shown above. But now, we can see a small bar is visible in the n = 7 bin: This approach to differentiating between empty bins and those with very low counts gives viewers a visual indication of where data exists, without drastically changing the look or interpretation of the overall distribution. ## **Choosing between continuous and discrete histograms** Histograms are often used to visualize the distribution of **continuous** quantitative variables like temperature, electricity usage, or file size. But, quantitative data can also be **discrete**, meaning that observations only exist at certain values. Often these are counts. For example, a political survey might ask participants how many children they have. The resulting data is discrete because values are limited to non-negative integers (0, 1, 2, etc.). With discrete quantitative values, it often makes more sense to show a bar for each unique value, rather than aggregating multiple values into broader bins like we do in continuous histograms. That’s especially true if there only exist a relatively small number of unique values. For example, the histogram below shows the distribution of household size (number of people) based on 41,569 survey responses in San Francisco: Discrete histograms visualize the frequency of each unique value in a separate bar, rather than binning counts across a continuous range, as shown here for household size in San Francisco. Data: [San Francisco City Survey Data](https://catalog.data.gov/dataset/san-francisco-city-survey-data). When we ask a BI visualization tool to create a histogram, however, it only sees a series of numbers. It has no sense of the methodology, or domain expertise, needed to determine whether the underlying variable is discrete or continuous. So a visualization tool can’t know for certain whether a histogram should be continuous or discrete. We can, however, make an educated guess about which is a better default option based on characteristics of the data. In canvases, we include some simple logic to determine whether we generate a continuous or discrete histogram. Our conditions for defaulting to discrete are as follows: - Fewer than 20 bins total - For each non-outlier bin, the minimum value is equal to the maximum value, ensuring that each bin represents a single value - Bins are not fractional With those rules in place, we can see the two different types of charts created in a canvas, depending on the selected data. In an Observable Canvas, a user first creates a large chart for a continuous quantitative variable (survey weight), which defaults to a continuous histogram based on the data. When they next create a large chart for a discrete quantitative variable (ranked responses for water infrastructure), a discrete histogram is generated instead. Data: [San Francisco City Survey Data](https://catalog.data.gov/dataset/san-francisco-city-survey-data). ## **Building a histogram with Observable Plot** Want to try building a histogram? With [Observable Plot](/content/plot/index.html), it only takes a few lines ofJavascript code to get started. Jump into an [Observable Notebook](/content/platform/notebooks/index.html), drop in your data, and search for “histogram” in the new cell menu, or copy and paste the code snippet below into a JavaScript cell. Once you’ve replaced the **dataset\_name** and **variable\_name** with your own, you’re on your way to a custom histogram! ``` Plot.plot({ marks: [\ Plot.rectY(dataset_name, Plot.binX({ y: "count" }, { x: "variable_name" })),\ Plot.ruleY([0])\ ] }) ``` ## **Explore more built-in charts in Observable Canvases** Histograms are extremely useful and common charts to explore data distributions for business analytics. In canvases, we’ve designed generalizable histograms that can give you a better first look at the bulk of your data, ensure that small bins aren’t hidden from view, and default to a discrete or continuous chart based on characteristics of the data. Learn how you can do fast, visual analytics in our new collaborative canvases, and [sign up for early access to canvases today.](/content/platform/canvases/index.html) \]\]>https://observablehq.com/blog/visual-data-analysis https://observablehq.com/blog/visual-data-analysisTue, 13 May 2025 00:00:00 GMTData practitioners know that a critical first step in [data analysis](/content/learn/what-is-data-analysis/index.html) is taking a careful look at their data in exploratory charts and tables. They also realize the value of building clear, compelling visuals for end products like dashboards and reports. So when it comes to keeping a close eye on their data, analysts have the bookends covered. But something unfortunate often happens in the in-between, when analysts are heads-down in data cleaning, [wrangling](/content/learn/what-is-data-wrangling/index.html), modeling, troubleshooting, and iterating: [data visualization](/content/learn/what-is-data-visualization/index.html) falls by the wayside. Visual data analysis merges data visualization and analysis, helping viewers to explore and answer questions about complex data through charts, tables, and graphics. That’s valuable throughout the entire analysis process, not just at the endpoints. Here, we share how end-to-end visual data analysis creates more opportunities for discovery, supports [cross-functional collaboration](/content/blog/collaborative-analytics/index.html), and reveals mistakes before too much damage is done. ## **Create opportunities for on-the-fly discovery** In the messy middle of data analysis, new patterns and anomalies can be revealed as you wrangle, aggregate, and analyze data in different ways. This can inspire questions and exploration paths that you hadn’t previously considered. If you’re not visualizing the data at each step, however, those discoveries can remain hidden in overwhelming tables, or worse — entirely unviewable, only existing ephemerally within an intermediate line of code. Manually building visualizations to explore the output of each data processing step can be prohibitively time consuming. Hence, the data visualization desert. A good alternative is to use tools that display or generate visual summaries at each step by default. [AI is an increasingly powerful option to draft fast, exploratory charts](/content/blog/three-ways-to-use-ai-for-exploratory-data-analysis/index.html), and can drastically cut the time and effort required to keep data visible throughout your data analysis. ## **Improve accessibility and interpretability** The data analysis process is often inaccessible to cross-functional collaborators like business and product managers who may lack familiarity with the programming languages or analysis tools used by your data team. And working openly, for example by sharing your code on GitHub, is a far cry from really empowering them to participate in your work. Nowadays, most people have at least a basic level of visual [data literacy](/content/learn/what-is-data-literacy/index.html). They’re constantly exposed to data visualizations in the news, through fitness tracking apps, while digging into unexpectedly high electricity bills, and more. Visualization has become somewhat of a universal language that allows many people — regardless of technical knowledge or skills — to make sense of data. This makes visual data analysis an accessible way to help non-coding collaborators review and interpret analyses, and give high quality feedback. When collaborators visually follow how data is transformed throughout analysis, they can confidently interpret what they’re viewing at any point. ## **Notice mistakes and anomalies early on** In business analytics, mistakes are both inevitable and potentially costly. Unit conversions get fumbled. Whole groups inadvertently disappear with the smallest typo (was it `== “blueWhale”`, or `== “bluewhale”`?). Oh, and that zip code? Yeah…at some point that got converted to a number, when it should have been treated as a nominal variable in your model. The types of mistakes described above won’t show up as error messages in the console, since code doesn’t know (or care) if what you’re asking it to do with the data is actually correct. If you’re lucky, they’ll be caught in code review by an eagle-eyed colleague, or while diagnosing obviously wrong outputs. Unfortunately, data processing mistakes don’t always reveal themselves so easily. Visualizing data throughout analysis is an additional way that you and your collaborators can catch mistakes. With more eyes on the data the whole way through, you’re more likely to identify issues with how it’s processed. And, since your team can visually track changes to the data at each step, it’s easier to determine exactly where in the analysis things went wrong. Visualizing data throughout your analysis increases the likelihood of finding mistakes and noticing anomalies (like missing data, as shown above) earlier in the process. Finding mistakes may sound like a bad thing. But, if it happens early enough in the process, it’s a gift. Visual data analysis can increase the likelihood of catching mistakes and revealing red flags, before it’s too late. ## **Don’t let data out of your sight** Manually building exploratory charts at each step in data analysis can be prohibitively time consuming, which means it often falls by the wayside. The result is a data visualization desert that keeps costly mistakes and valuable insights in the dark. End-to-end visual data analysis helps you to keep eyes on the data throughout your analysis, ensuring that surprises — whether exciting, or concerning  — don’t go unnoticed. By using tools that make it quick and easy to see your data throughout the analysis process, instead of just at the bookends, you can maximize [visibility](/content/blog/how-to-build-data-culture#increase-the-visibility-of-data-analyses/index.html) without slowing down for more accurate and interpretable insights. Want to learn more about best practices for modern data exploration and analysis? Check out other recent posts on the Observable blog, and [sign up for our newsletter](/content/home?modal=newsletter/index.html) to see new resources as they come out: - [4 ways to avoid common gotchas in data analysis](/content/blog/4-ways-to-avoid-common-gotchas-in-data-analysis/index.html) - [Move faster, together, with collaborative analytics](/content/blog/collaborative-analytics/index.html) - [Three ways to use AI for EDA](/content/blog/three-ways-to-use-ai-for-exploratory-data-analysis/index.html) \]\]>https://observablehq.com/blog/collaborative-analytics https://observablehq.com/blog/collaborative-analyticsThu, 08 May 2025 00:00:00 GMTCollaborative analytics brings data analysts, engineers, domain experts, and business managers into the data lifecycle to answer questions together. By involving stakeholders throughout the analysis process, data analysts can better focus their efforts, reduce tedious back-and-forths, and help teams converge on informed decisions, faster. In this post we introduce collaborative analytics, describe how it makes [data analysis](/content/learn/what-is-data-analysis/index.html) better and more efficient, and share how to put it into practice at your organization by asking the right questions and using the right data analysis tools. ## **What is collaborative analytics?** Collaborative analytics is the practice of working with team members and stakeholders throughout the analysis process from ideation to final data product. Who should be involved, and to what extent, depends entirely on the specific project at hand. In the business world, this often includes: - **Stakeholders and business managers**, who (1) help define problems and questions so that analysts can better focus their work and reduce wasted effort, and (2) help envision and outline end products so that analysts can work with a specific endpoint in mind, reducing thrash and work left on the cutting room floor - **Data analysts,** who dig into the data to find insights and answers to stakeholder questions through [data exploration](/content/learn/what-is-data-exploration/index.html), visualization, and analysis - **Domain experts,** who share critical context and knowledge about data and methods, so that analysts can approach questions and interpret findings responsibly and with confidence - **Data engineers,** who streamline data pipelines and sharing, ensuring that analysts and collaborators can quickly access and work with data from different sources [Collaborative analytics](/content/learn/glossary#collaborative-analytics/index.html) democratizes data work and involves stakeholders throughout a project, instead of just tapping them for a random no-context stats question or copy edits on a final report. That brings valuable perspectives and expertise to the entirety of your analysis, which benefits efficiency, accuracy, and impact. ## **Go faster, together** A well-known saying (of [questionable attribution](https://andrewwhitby.com/2020/12/25/if-you-want-to-go-fast/)) starts: “If you want to go fast, go alone.” But that’s not always true in business analytics, where speeding off solo can come at a cost once external feedback catches up with you. For example, it’s inefficient to develop an entire dashboard in isolation, only to later find that you’ve built something that already exists, or that no one actually needs. And, it’ll cost you time in the long run to barrel through analysis, only for a business manager to later share a major flaw in how you’re thinking about a method or metric. Collaborative analytics can help businesses increase efficiency in a number of ways. By getting input and feedback from coworkers and stakeholders from the start, data teams can focus their analyses on questions that will be most useful for final users. Shared, transparent discussions can bring to light previous efforts on the same (or related) questions, helping analysts to avoid redundant work that others have already tried. And keeping stakeholders on the same page, where they can all see discussions and decisions, minimizes unnecessary back-and-forths and unexpected surprises at the tail end of a project. Even when it comes to decision-making, collaborative analytics is a win for efficiency. When business managers and stakeholders have a solid understanding of the analysis, intermediate decisions, and revisions made along the way, you can avoid rehashing old questions or realizing major points of contention right at the end. That helps your team converge on decisions much more quickly. So, we recommend erring on the side of being a collaborative tortoise, instead of a lone hare. Taking the time to include collaborators throughout analysis can avoid unused or redundant effort, minimize surprise requests, and speed up decision-making to get you all to the finish line, faster. ## **Strategies for better collaborative analytics** It’s easy to say “make sure you include folks throughout the data analysis process!” In reality, meaningful collaboration takes planning, effort, and careful project and stakeholder management. Here, we share strategies to bring better collaborative analytics to your organization by asking the right questions up front and adopting tools that empower your collaborators to participate in the work. ### **Invest in history lessons** Time is precious, reinventing the wheel is a drag, and collaborators often have useful knowledge based on prior projects that can spare you unnecessary research and testing. So, it’s worth taking the time to learn what’s already been done. Here are some questions to ask yourself, and your teammates, as you get started: - **To avoid doing redundant or unnecessary work, ask:** Do the answers to these requests already exist somewhere? If yes, can we iterate on the existing solution, instead of building from scratch? - **To build on knowledge and experience of your team, ask:** Who on the team has worked on similar analyses? Can they share notes and artifacts from their work? What challenges or failures did they experience along the way, and how did they overcome them? - **To understand context and motivation, ask:** Why hasn’t this analysis been done before? Why is it a priority now? This is not a comprehensive list, but starting with those questions can help you avoid wasting time on work that has already been done while building on the knowledge and experience of your team. Plus, this process will likely reveal people who you’ll want to involve as collaborators. ### **Use tools that let collaborators engage directly with the analysis** When data work is separated from discussions around it, there is potential for confusion and crossed wires. Let’s say a data analyst is developing a dashboard in Python. They share drafts of the rendered page with the VP of Sales. That VP might give feedback in a private message on Slack, trying to describe in words what they’re referring to (for example: “In the chart showing MRR there is a drop in January that is an artifact of our pricing change. We should point that out.”). This is a common, but potentially problematic, interaction. It forces the VP to describe the part of analysis they’re talking about in words, which can be difficult and ambiguous (You: “Do you mean the total MRR chart, or the MRR by product breakdown?”) and lead to crossed wires that make extra work (You: “I’ve added annotation to note the date of the price change.” Them: “Oh I meant for January 2022, not 2021.” You: “🤦”). A better option is to use tools that keep the analysis, and discussions around it, all in one place. This approach streamlines the process of giving and acting on feedback, and leaves less room for misinterpretation. ### **Show your work** When non-coding collaborators only have visibility into the bookends of data analysis (like the raw data, and a final chart or report), it limits entry points where they can ask questions or realize new opportunities for exploration and analyses. Without being able to track data from database to dashboard, it can be difficult to closely review the work and confidently interpret findings. [Visual analysis](/content/blog/visual-data-analysis/index.html) leans heavily on interactive charts that help collaborators track data as it is transformed from raw data to final display. This gives stakeholders the context they need to responsibly inspect and consider the analysis at each step, for more meaningful feedback and more trusted results. ## **Learn more** When data practitioners involve stakeholders throughout the analysis process, build on existing knowledge, and use tools that foster collaboration, they can get from raw data to trusted business decisions, faster. Want to start democratizing data projects, and bring more people to the data table? Check out [Observable Notebooks](/content/platform/notebooks/index.html): our reactive JavaScript notebooks where teams can explore, analyze, and visualize data together, while building on examples from the Observable community. \]\]>https://observablehq.com/blog/why-choose-observable-bi https://observablehq.com/blog/why-choose-observable-biThu, 01 May 2025 00:00:00 GMTObservable has a lot to offer. The fast and flexible two-dimensional way of working on Observable Canvases. The built-in real-time collaboration. The powerful visualization capabilities. All built on a modern, web-first platform. Read on as we lay out the many reasons for choosing Observable over legacy BI tools. ## **A new level of exploration** Escape the linear workbook model and [explore your data on a two-dimensional collaborative canvas](/content/platform/canvases/index.html). Build a mind map of your thought processes, analysis steps, and results. Never lose track of what you’ve already explored, go back to previous thoughts and expand on them 
at any time. Data analysis involves branching paths that transform and summarize tables in different ways. Zoom in and out of your analysis to work on a detailed visualization, or see the entire flow on one screen. Get powerful guidance that keeps track 
of what you’ve already done and understands your data, in the same 
space you’re working in. Stay in the flow of analysis at all times. ## **Native, real-time collaboration for faster decisions** Why work alone and send messages back and forth when you can work together, in the same space, in real time? Ask questions, get help from others, and help others out. See what everyone else is seeing, no matter where you are. Use powerful tools together, and build better analyses and visualizations as a team. ## **One place for your data and visuals** Rather than having to switch between modes, pages, or even tools, perform all your analytics reporting work in one place. Explore your data as tables, filter and join them, then use them to create powerful visualizations. See and follow the steps that lead from your source data to your analysis products, and easily understand what others have built by following theirs. ## **Visualizations, not just charts** Yes, we all love a good bar chart. But many situations call for more advanced and specific charts and visualizations, the kinds that other tools don’t have at all, or only as opaque, isolated modules. Build better, more expressive, more useful visualizations on the canvas. Get recommendations for more elaborate and specific visualizations, based on the latest research and decades of experience using data visualization to solve real problems. Create richer, more powerful visualizations than ever before. Explore the endless power and possibilities of [Sankey diagrams](/content/learn/glossary#sankey-diagram/index.html), waffle charts, treemaps, advanced maps, and many more. Sankey diagrams show flows through a network, like power from generation to distribution and use. Need to shape your data first? 
Use SQL nodes to write and reuse queries directly on the canvas — no switching tools, no hidden steps. Whether you’re cleaning data, filtering, [joining](/content/learn/glossary#joining-tables/index.html), or transforming; your queries stay visible, editable, and part of the larger story 
your analysis is telling. ## **Real-time collaboration for faster decisions** Bring everyone into the same space 
and collaborate in real time on a 
shared canvas that feels like a digital whiteboard. Sketch out ideas, annotate visualizations, and leave comments right next to the data. See what your teammates are seeing, no matter where you are. Use powerful features like annotations, stickies, and highlighting together to build better analyses and visualizations as a team. ## **A higher ceiling with SQL and code** Even the most powerful user interface can only get you so far. Write your own queries using the language of databases, SQL – either from scratch, based on what you’ve created using Observable’s canvas tools, or with a head start from our integrated AI. When you’re ready to share a more polished view, turn your canvas into a dashboard with a few clicks, no extra tools or rework required. Share insights with your team and keep everyone aligned. And because it’s your work, you can take it with you. Export your work in many different formats (PNG, SVG), or your data (CSV, JSON) in whatever format your team needs. ## **See it with your own eyes!** Whether it is the real-time collaboration, the single place for your entire analysis, the unparalleled visualization features, or the flexibility of using code, no code, or both – Observable is the BI solution for your needs today. Curious to see all we have to offer for yourself? Request a demo today! \]\]>https://observablehq.com/blog/how-to-query-databases-data-analysis https://observablehq.com/blog/how-to-query-databases-data-analysisMon, 21 Apr 2025 00:00:00 GMTDatabases allow you to both store your data and answer a variety of questions about your data. In a previous post, we covered [how database tables are structured and why databases look different from spreadsheets](/content/blog/databases-101-basics-data-analysts/index.html). In this post, we’ll take a closer look at the kinds of questions you can ask of a single table, and start to analyze our data. We’ll start with simple questions like counting the number of rows, and then move on to more aggregations like sums and averages, and how to compute those for different groups in the data. At the end, we’ll also cover filtering data to select rows by a criterion. SQL, the [_Structured Query Language_](/content/documentation/cells/sql/index.html) (usually pronounced “sequel”), is the main way to talk to relational databases. Being able to write some basic SQL queries, or at least read and understand them, is a useful skill for anyone working with data. It also seems a bit silly to talk about database queries and not show the code, which is why we’re including it here. The goal of this post is not to teach you the intricacies of SQL. SQL can get pretty involved, so in this post we only cover simple queries that are quite straightforward. If you don’t care about SQL, you can skip them and still learn about the basic ideas. ## **Let’s count!** To start, we’ll build the simplest query to count our records in our database. We’ll be using the same table of [orders at a fictional pizza chain](/content/@observablehq/pizza-paradise-data/index.html) as last time, which looks like this: How many orders do we have in our dataset? To find out, we’ll write a SQL query that performs a count operation. Simple SQL queries generally look like “SELECT FROM ”, and we’re using the COUNT() function here to perform the count. Since SQL is an old-school language from the 1980s, it’s common to capitalize all SQL keywords, which I’ll do here for clarity. You don’t have to do that if you feel like that’s too shouty, though. We can also give the result of the COUNT() function a name so we can use it for other purposes later, that’s what the “AS numOrders” part is for. What are we selecting from? The pizzaOrders table! Since this is the result of a database query, it is shown as a table, even if we’re only getting a single value back. ``` SELECT COUNT(*) AS numOrders FROM pizzaOrders ``` So now we know that we have about two million records in total, across all our locations. How many stores do we even have? You may be wondering about the asterisk in “COUNT(\*),” which basically means “count everything.” Instead, we can count specific things, like the number of unique store IDs. SQL calls this DISTINCT, so instead of COUNT(\*), we can use COUNT(DISTINCT storeId) to get the number of stores. ``` SELECT COUNT(DISTINCT storeId) AS numStores FROM orders ``` We can do the exact same thing with the customer ID to find out how many customers we’ve had: ``` SELECT COUNT(DISTINCT customerId) AS numCustomers FROM orders ``` _Now that’s great,_ I hear you say, _but what if I want to know how many sales were made by each store?_ To find out, we can group the results by the store ID, so we don’t just get a single count, but instead the count for total sales _for each store_. We’ll go back to using COUNT(\*) here to count all the sales, but now we’ll include the storeId in the query, then use a GROUP BY clause to create separate counts for each store. ``` SELECT storeId, COUNT(*) AS numOrders FROM orders GROUP BY storeId ``` So now instead of a single value as the response, we get a table. It shows the store IDs in one column, and the order counts in the other. We call this _aggregation_ because we’ve reduced the total ~2 million records for each order to just 32, one for each store. The table here is just showing the first ten results, and you might have noticed that there doesn’t seem to be an obvious ordering. It’s not by ID, and it’s not by count. This goes back to what I said in [the previous post](/content/blog/databases-101-basics-data-analysts/index.html) about the order of rows or relations in a table not being fixed. When you perform a query, there’s no way to know what order the results will come out in. We can, however, specify how we want the results to be sorted using an ORDER BY clause. This works the same as GROUP BY, in that we can specify any of the columns in the query as the sort criterion – including ones we’ve created in the query, like _numOrders_. However, by default, ORDER BY sorts in ascending order, from the smallest to the largest numbers. You often want descending order to see the largest numbers or highest counts, which is done by adding DESC just after the column name you’re sorting on. ``` SELECT storeId, COUNT(*) AS numOrders FROM orders GROUP BY storeId ORDER BY numOrders DESC ``` ## **Computing totals and averages** If that was your first experience with SQL, just counting was a lot! But the good news is that computing other values is very similar, so we can use the same patterns to do that. First, let’s see what the total revenue of all our sales was for the time period covered in this table. Instead of COUNT(), we’ll use SUM() to add up order prices from the _total_ column, and everything else stays the same. ``` SELECT SUM(total) AS ordersTotal FROM orders ``` As before, we can break this down by store and sort in descending order, using GROUP BY and ORDER BY. ``` SELECT SUM(total) AS ordersTotal FROM orders GROUP BY storeId ORDER BY ordersTotal DESC ``` If you compare this to the final example in the section above, the only differences are that we’re using SUM instead of COUNT, and the name of the sum is different. Everything else is the same, since the structure of the query is the same: break down the table by store ID, then compute the sum (in this case, or the count above) for each, and then sort the results by the computed value. There are many functions other than SUM to perform common operations, such as AVG for the average, and MIN and MAX. For example, we found the total sales for each store above. But what time period does our dataset even cover? We can find out with this simple query that selects the minimum and maximum order dates, and calls them first and last, respectively. ``` SELECT min(orderDate) AS first, max(orderDate) AS last FROM orders ``` This kind of simple minimum and maximum query is also useful [to look for outliers or special values](/content/blog/data-exploration-checklist/index.html). There are also functions to aggregate dates so we could compute sales by month or year in addition to (or instead of) the stores. ## **Filtering data** In addition to aggregating data, another operation databases do really well is filtering, which is done in SQL by adding a WHERE clause containing your condition. For example, the query below returns only orders over $100. ``` SELECT * FROM orders WHERE total > 100 ``` Again, we’re getting an unsorted list of orders by default, and perhaps we don’t even want to see each individual order – we may just want to find out how many orders exceeded $100. In that case, we’d want to count them by adding a COUNT clause. ``` SELECT COUNT(*) AS numOrders FROM orders WHERE total > 100 ``` This clause can also be combined with the other pieces we’ve already used, for example to count how many orders over $100 were recorded by each store, and then sort the stores by that count. We use the same aggregation (GROUP BY) and sort (ORDER BY) clauses, and just add a WHERE clause. The WHERE applies _before_ the aggregations, so this filters the data to only those rows with a total of over 100, and then does the counting from there. ``` SELECT storeId, COUNT(*) AS numOrders FROM orders WHERE total > 100 GROUP BY storeId ORDER BY numOrders DESC ``` Filtering is incredibly important when doing [data analysis](/content/learn/what-is-data-analysis/index.html) to exclude special (or erroneous) values, and to restrict the query to a known time range or subset. ## **Unleash the power of databases!** There is a lot more that can be done here of course, but this should give you a sense of what is possible when analyzing [databases](/content/learn/glossary#database/index.html). Queries can be combined to ask a variety of different questions, and we’ve only even used a single table so far. AI is making it even easier to build and edit more complex queries with text-to-SQL. Learn how to responsibly use AI for fast exploratory data analysis, while avoiding common pitfalls: - [Three ways to use AI for exploratory data analysis](/content/blog/three-ways-to-use-ai-for-exploratory-data-analysis/index.html) - [4 ways to avoid common gotchas in data analysis](/content/blog/4-ways-to-avoid-common-gotchas-in-data-analysis/index.html) \]\]>https://observablehq.com/blog/introducing-canvases-early-access https://observablehq.com/blog/introducing-canvases-early-accessTue, 15 Apr 2025 00:00:00 GMTToday we’re excited to announce a breakthrough in data collaboration: [**Observable Canvases**](/content/platform/canvases/index.html) **.** Data analysis involves branching paths that transform and summarize tables in different ways. Canvases are computational whiteboards for exploring, visualizing, and analyzing data. By combining queries, tables, charts, freehand sketches, highlights, notes, and more — augmented with contextual AI that’s designed to be more interpretable and fluid — canvases capture and communicate context and help everyone on your team to engage with analysis. ## Why canvases [Data analysis](/content/learn/what-is-data-analysis/index.html) requires constantly looking at data to make sure we’re on the right track. We peek at the first few rows of a table, check the extent of a column, or count missing values. Spot checking is essential to catch surprises and avoid bogus results (or: _garbage in, garbage out_). Like wiring up a circuit on a breadboard, we test components and connections as we go. And like any creative activity, analysis is not a straight line from question to answer. It is exploration, experimentation, and iteration. Vague and underspecified questions require us to establish precise, quantitative definitions — of _product activity, sign-up funnel, top customers,_ and the like. We try one approach and then another, advancing and retreating, pursuing serendipitous discoveries. Our quarry is not merely a private insight but something that can be shared — then inspected, tested, poked, validated — with our stakeholders. To help an organization operate intelligently, we must externalize assumed context, annotating charts and numbers so they can speak for themselves. But how well do our tools actually support this process? Too often, today’s tools don’t proactively surface visual summaries, instead requiring the analyst to spot check manually. Too often, tools force a linear workflow that penalizes experimentation and flattens thinking. And too often, tools don’t capture the messy context and thinking around analysis, leading to results that don’t communicate. We want to fix that. ## A better medium for observing data Canvases let you arrange analysis spatially. Starting from a table in your data warehouse, you can add downstream nodes to filter or group rows, to join tables, or to show charts or summary metrics. Having two dimensions to organize your work increases flexibility. It’s easier to keep track of what you’re doing, and for others to see it at a glance: just zoom out to see the big picture. And it gives you space to try out ideas without disruption, whether you’re working alone or together. Simply option-drag, Command-D, or copy-paste to duplicate nodes. Summary tables reveal patterns in your data by grouping values and showing distributions at a glance. Canvases use summary tables to help you perceive data throughout analysis. These visual overviews help you notice anomalies and interpret insights. To accelerate [data exploration](/content/learn/what-is-data-exploration/index.html), canvases encourage direct manipulation — just click on data in a table or chart to filter, exclude, derive, sort, and the like. Nodes can be constructed either visually, by pointing and clicking, or by writing code. Canvases work performantly with large data by pushing compute to the underlying data source. Instead of materializing the full results of queries, canvases compute fast column summaries, say, to bin quantitative values for a histogram or to group the top categorical values for a bar chart. Any node on the canvas can be an implicit common table expression (or CTE) in a SQL query. (It’s query pushdowns all the way down.) And canvases capture query snapshots as you work, so everyone will see the same results — whether today, tomorrow, next week, or next year. Draw directly on your charts to call out insights, connect ideas, or add context as you go. While it may seem counterintuitive, we want canvases to feel a bit _messy_ — because analysis itself is messy. Canvases support freeform whiteboarding powered by [tldraw](https://tldraw.dev/). See something interesting in a chart? Simply draw on it, highlight it, or slap an arrow and label on it. By annotating as you go, you capture context and your work communicates better by default. Your stakeholders can ask questions and engage directly on the canvas for faster, deeper feedback. With realtime multiplayer editing, cursor chat, following, and more, the canvas instantly becomes a living, shared screen. Invite stakeholders in to explore, comment, and contribute — all in a shared, live workspace. And of course, AI. We’re excited about the potential of the canvas to make AI a stronger, more reliable partner for analysis. LLMs are powerful yet unpredictable and prone to mistakes — not something you want when making critical business decisions. By integrating AI directly into the canvas, by using spatial layout and visual summaries, we seek to make AI-driven analysis more interpretable, inspectable, and steerable. We want AI to participate in a fashion similar to a human collaborator, leveraging context, annotating results, and even using multimodal AI to “see” charts. AI is no substitute for humans, but it can make us more efficient, creative, and autonomous. ## A space for cross-functional collaboration For years, business users have dreamed of “self-serve” analytics where they can answer their own questions with just a few clicks; but too often, business users are relegated to specialized “self-serve” tools that isolate them from data practitioners and offer only limited capabilities. This results in inconsistent and incorrect results, questionable interpretations, missed insights, and ultimately a loss of trust in data. Isolating roles is counterproductive; we should instead bring people together more efficiently. Analysis can be presented visually, spatially, and contextually on the canvas to aid interpretation. And stakeholders can engage directly, increasing agency and removing ambiguity when providing feedback. Data analysis becomes more fluid and efficient when people work together. ## Early access starting today We’re excited to announce our early access program for [Observable Canvases](/content/platform/canvases/index.html) today. If your data team is interested in participating, or you’d like to receive development updates, please [let us know](/content/platform/canvases?modal=demo/index.html). We’d love your feedback. Canvases will be generally available later this year. _Update: Canvases is now in_ [_general availability_](/content/blog/canvases-summer-release-2025/index.html) _. Get started_ [_here_](/content/get-started/index.html) _._ \]\]>https://observablehq.com/blog/databases-101-basics-data-analysts https://observablehq.com/blog/databases-101-basics-data-analystsThu, 10 Apr 2025 00:00:00 GMTDatabases are the main way that large organizations store data that needs to be analyzed. But databases don’t just store the data, they also allow you to work with it: ask questions by running queries, create new combinations of facts by [joining tables](/content/learn/glossary#joining-tables/index.html), all to understand what is in the data and to better inform business decisions. In this post we cover some basics of databases including core terminology, an introduction to tables (and how data is structured in them), and the basics of keys and joins. We’ll draw some comparisons to spreadsheets, since that is the other common format in which people encounter data. This initial post will set you up to dive into our upcoming blog posts on basic [data analysis](/content/learn/what-is-data-analysis/index.html) in databases using queries and joins. ## **What are relational databases?** Most databases you will encounter in business contexts are called _relational_ databases, hence also the slightly old-school acronym RDMS for _relational database management system_ (also sometimes called RDBMS, using the two letters DB for database). But what exactly does it mean for a database to be relational? In a relational database, data is stored in tables. This is much like in Excel, however the format between the tables is quite different. Each row in a database table is treated as an independent piece of data, and is in itself a relation: it combines a set of attributes (in the columns) that taken together describe one thing or data item (like a sale). The table of pizza orders shown below is from [our Pizza Paradise dataset](/content/@observablehq/pizza-paradise-data/index.html). The IDs in the first three columns might look cryptic at first, but I picked this example on purpose. It’s a better representation of the kinds of data tables you’ll likely encounter than a nicer-looking example. This table contains orders, and for each it has an identifier for the order (which is also the key for this table, we’ll get to that below), IDs for the customer and store involved in the order, an order date, a number of items, and a total amount. Each row here is a relation, tying together the various attributes (like customer IDs, order date, etc.) with the order ID. Rows are identified not by their position in the table, but by a key. In this case, the key is simply the order ID in the first column, but it can be more complex. We'll talk a little bit more about keys below. In Excel, you’re probably used to defining operations between rows, where you can rely on the ordering being stable. In databases, by contrast, the ordering of rows is completely arbitrary, and not fixed. In the example above, it appears to be based on order ID, but we could just as well order by date or total amount. This more flexible data model also allows for much easier aggregation operations to get totals by store, or by month, but we’ll get to those in a future post. ## **Table shapes and normal forms** Databases also differ from spreadsheets in their shape. Spreadsheets are usually organized in a two-dimensional way, where one dimension (like the month) goes across the columns, and another (like the year) across the rows. This kind of organization makes sense in a spreadsheet, where it’s often interesting to compute values (like sums or averages) across both rows and columns: sum of sales across a year, average sales over the months of a year. Database tables, by contrast, are usually organized in a way that favors more rows over more columns. This is sometimes called the “long and skinny” format (as opposed to the “short and wide” of spreadsheets), or more formally a _normal form_. In statistics, this is also known as [tidy data](https://tidyr.tidyverse.org/articles/tidy-data.html). Instead of a column for each month and a row for each year, a database table has columns for the year and month, and then rows for each combination of the two. The same would be true if we were looking at products: instead of a column for each product, there would be a single column called ‘product’ containing the names of the products, requiring many more rows. This data structure is the core of the relational model. We end up with a lot more rows, but this way allows us to organize data not just in two dimensions, but over a much larger number of dimensions and for as many attributes as we want. ## **The difference between fact tables and dimension tables** Let’s look at the pizza orders example from the top of this post again. I mentioned that it looks kind of cryptic with all these IDs. Who exactly is customer C505924? And where would you find store S948821? While this might seem odd at first, it is the consequence of a core organizational principle of databases in normal form: data is split into _fact tables_ and _dimension tables_. The example above is a fact table, which connects a number of different types of information (customers, stores, and order data). The details of each customer and store live in dimension tables, which are linked through their respective IDs. The advantage of separating data into fact and dimension tables is that we have to keep each customer and store’s information only once in the dimension tables, rather than hundreds or thousands of times as attributes in the fact table. That system also gives us more consistency, because we won’t suddenly find the same store with different addresses or ZIP codes. ## **Understanding database keys and joins** The way fact and dimension tables are connected is through _keys_. We’re not going to cover the joining of tables in this post, but it’s important to have an idea of the basic concept to understand why databases are organized like this. A key is generally a unique attribute (or set of attributes) that allows a database to find a particular row in a table. The easiest way to do this is to assign each row a unique identifier, like the order ID in the first column in the pizza orders example above. The key can also be a combination of multiple values. A join then allows us to create a new table by looking up the IDs that are found in a fact table, and inserting the information from the dimension table instead. So again using the pizza orders, the database would look up which store has ID = S948821 and insert its address, and look up which customer has ID = C505924 and insert the name. That might then give us the information to print out a receipt. Or we could look up the customer’s age group and the store’s ZIP code, and count how many times each combination occurs. That might tell us something about the different customer demographics in different areas. If this all seems confusing, don’t worry. We’ll get into joins in much more detail in the coming posts. ## **Databases are your friends!** Databases might seem difficult and unfamiliar at first, but like spreadsheets and simple data files, they just require some understanding of core concepts I covered here like “long and skinny” data, keys, and types of tables. Most data lives in databases, certainly when it comes to larger amounts of data, so knowing your way around them is always going to be a plus for [data analysis.](/content/learn/what-is-data-analysis/index.html) Come back soon! We’ll get into more detail about how table joins and data transformation in databases work in upcoming posts. \]\]>https://observablehq.com/blog/ai-tools-llms-d3-code https://observablehq.com/blog/ai-tools-llms-d3-codeThu, 03 Apr 2025 00:00:00 GMTThere’s a lot of interest in (and frankly, hype around) AI right now. In particular, large language models (LLMs) are getting a lot of attention, for reasons both good and bad. They can be incredibly useful, but they also sometimes hallucinate information, and it’s not always obvious what is true and what isn’t. One area where I have found AI assistants to be quite useful and capable is in writing code. Coding seems to be particularly well covered in most mainstream LLMs’ training data, and it’s relatively easy to verify that the code does what it’s supposed to do — at least on a basic level. There are many caveats here, and for code in production use there would need to be extensive testing. There’s also a clear danger in [using code without fully understanding it](https://arstechnica.com/ai/2025/03/is-vibe-coding-with-ai-gnarly-or-reckless-maybe-some-of-both/). But for our purposes of writing code to generate charts and data visualizations, AI seems well-suited for this task. So I decided to try asking [Anthropic’s Claude](https://claude.ai/) Sonnet 3.7 to help me create some basic charts and data visualizations in [D3](https://d3js.org/). ## **Making a simple line chart with AI** First, I attached a CSV file with a dozen lines of data from our [Apple stock price demo dataset](/content/@observablehq/sample-datasets?collection=@observablehq/getting-data-in-and-out/index.html). It contains data about Apple’s stock price from 2013 to 2018, in the form of opening and closing prices, volume, etc. I then gave Claude this prompt: > _write d3 code that plots the closing price from the attached csv file over time as a line chart_ It produced a complete HTML file that loaded D3 (though a slightly older version, 7.8.5, despite 7.9.0 having been out for almost exactly a year at this point), included my demo data, and generated all the code to produce this chart: It’s a fair amount of code, so instead of showing it all here, I’ve added it to a [github repo](https://github.com/observablehq/blog-llm-code-assist). But it’s perfectly reasonable, well-structured JavaScript code calling D3 functions, the way you’d write it yourself in a plain HTML file. If I had given it more context about wanting it inside a React or [Observable Framework app](/content/framework/index.html), it would probably have produced the appropriate code. It’s interesting that it has made some assumptions here about the size of the dataset (since I didn’t say that it was a sample) and has included dots for each data point. It has also recognized that the data is all from a single month and put that in the title. If we add in all the data from our dataset, this gets a little messy: It’s easy enough to manually remove the points with a small adjustment to the code, to get this very acceptable line chart: I just removed the circle marks here, which also means losing the mouse-over tooltip. I later prompted Claude to do it for me, but I’ll come back to that at the end. ## **Adding quarterly averages** A single line chart in D3 consists of a fair amount of boilerplate code (the entire file for the chart above was around 300 lines), but it’s all quite straightforward. What if we wanted to get something a little bit more interesting, for example by adding quarterly prices? This is what I asked Claude: > _now group the prices by quarter_ And this is the chart Claude generated: It’s not very impressive with the small amount of data, but it works in principle. If I add in the rest of the data, and then remove the points and the quarterly text labels (which overlap and get messy), we end up with a decent chart: It’s not actually what I wanted (I had imagined [it would create bars](/content/blog/bars-vs-lines-time-series-data/index.html)), but we’ll get to that. The code is otherwise unremarkable, but in a good way. I question the choice of a dotted line for the quarterly averages, but that’s aesthetics. The only thing that stands out in the code is this rather elaborate way of computing the quarter from the month using a series of if statements. ``` let quarter; if (month >= 0 && month <= 2) quarter = 1; else if (month >= 3 && month <= 5) quarter = 2; else if (month >= 6 && month <= 8) quarter = 3; else quarter = 4; ``` Since the month in JavaScript is zero-based, there’s a much faster way to do it with a little bit of math: ``` const quarter = Math.floor(month/3) + 1; ``` Not really a big deal, but I’m curious where this comes from. I haven’t mentioned yet that Claude doesn’t just produce code, but gives fairly extensive explanations about what the code is doing and what assumptions it’s making. They’re quite interesting and help put the code into context. Claude also adds comments directly into the code. ## **Make it a bar chart!** These line charts are nice and all, but I really wanted a bar chart. Switching chart types is more work in D3 than, say, [Observable Plot](/content/plot/index.html), and a good use for some help from AI. So I asked Claude help me switch the line chart to a bar chart: > _turn it into a quarterly bar chart. Assume that there are several years of data, not just what's included here._ And here is the chart the AI assistant produced: Amusingly, Claude added code to generate more data in response to being told to assume that there’s more data. It even does it by quarter, generating 3-5 data points for each, which it appends to the original data I supplied, and then averages over everything by quarter. It doesn’t seem to know how many data points there should be in a quarter, but on the other hand it also doesn’t really matter since we’re only looking at the averages in the end. It’s also interesting that it doesn’t just generate random numbers, but adds in an upward trend. Here’s the relevant piece of code, including the comment it generated. ``` // Base the random close value on the original data range (60-65) // but add some yearly trend and variance const yearFactor = (year - 2013) * 8; const randomClose = (60 + Math.random() * 5 + yearFactor).toFixed(6); ``` We can replace the generated data with the real data, and get a serviceable bar chart. It could use a bit more space at the bottom, and the chart title doesn’t respond to the actual date range. But everything else works. This also shows the tooltip Claude has created. It’s interesting that it felt the need to include the number of data points, which usually wouldn’t be of interest (though they will differ slightly between quarters). ## **Redoing the line chart with a path** While I prefer the bar chart, it did bug me that I had to fix the first line chart myself. So I decided to test how well it can deal with references to earlier work and asked it to do the work for me instead: > _remove the dots from the first chart_ Here’s the result: Interestingly, it decided to smooth the line this time. If we add in all the data again, we end up with another perfectly fine line chart. It’s worth noting here that the way the x axis adapts is due to D3’s labeling functions, not the code generated by Claude. It would also be pretty easy to make the title respond to the date range covered by the data. A more elaborate prompt, or a follow-up, should get the LLM to create the relevant code. ## **Conclusion: AI is useful for writing D3 code** This is a very simple example, but it shows how LLMs can write a fair amount of D3 code that does what it’s supposed to do. Each of these resulting files is around 300 lines long, and they all worked without modification. It’s also interesting that it immediately understood what I meant by “d3 code” and produced an entire HTML file. This is undoubtedly due to the large amount of training data for D3 charts out there. There are some unusual choices, like the way quarters are computed, but they work just fine. Some of Claude’s other choices (like the inclusion of dots on the line chart) are probably due to the small data sample I provided. They’re easily changed, though. My prompts have also been quite short and not very specific. For example, the second chart could have easily been a bar chart if I had asked for it. There are also a number of [prompting techniques](https://huggingface.co/docs/transformers/tasks/prompting#advanced-prompting-techniques) that could help get the LLM to produce what I want, such as chain-of-thought prompting. LLMs and AI tools for coding have already proven themselves to be very useful and can save software engineers a lot of time. AI tools won’t do your [data analysis](/content/learn/what-is-data-analysis/index.html) for you, but they can help speed up repetitive tasks and code that is heavy on boilerplate – like writing D3 code for common chart types. \]\]>https://observablehq.com/blog/treemaps-history https://observablehq.com/blog/treemaps-historyThu, 27 Mar 2025 00:00:00 GMT[Treemaps](/content/learn/glossary#treemap/index.html) are a fairly popular [data visualization](/content/learn/what-is-data-visualization/index.html) technique, used to show data that can be broken down into categories. They’re popular in [business intelligence](/content/learn/what-is-business-intelligence/index.html) dashboards, often serving as overview charts that enable quick comparisons between groups. While they’re commonly used for business analytics data today, the original purpose and visual design of treemaps was actually quite different. In this blog post, we’ll briefly look at what treemaps are, where they originated, how they’ve changed over time, and how they’re used for BI and data visualization today. ## **What is a treemap?** A treemap subdivides space to show the proportions of a value as part of the whole. For example, if breaking down sales by department and product, the area of each rectangle would show the amount of sales for each, as a fraction of the total sales shown in the chart. The following example shows the breakdown of orders in our [Pizza Paradise demo dataset](/content/@observablehq/pizza-paradise-data/index.html). Color represents the pizza category, which is also the top level of the hierarchy. Each category is then subdivided by type of pizza, with the rectangles showing the number of orders for each. This modern use of treemaps is quite different from the historical origin of the treemap, however. ## **The original tree-map** The idea of the treemap goes back to 1990, when Ben Shneiderman was looking into ways to show [who was using how much space on a shared 80MB hard disk](https://www.cs.umd.edu/hcil/treemap-history/) at his research lab at the University of Maryland. He came up with a new way of subdividing space to reflect the relative sizes of files and directories, [which he called a _tree-map_](https://dl.acm.org/doi/abs/10.1145/102377.115768). It shows folders for different years, which contain files of different types. The areas of each rectangle reflects the size of the file or folder it represents. Some of the folders also have subfolders, like _1994_ with _DQtutorial_ in the visualization below: The _tree_ in tree-map is important because it initially was conceived as a way to show hierarchies. In computer science, hierarchies are called trees because they grow from a single root node and branch out into more nodes on each level. When looking at a file system, the size of each folder (or directory) is the sum of the sizes of the files and folders within it. The tree-map reflects this by allocating an amount of space to each that is the same fraction of the total space as the folder is of its parent. The strength of the tree-map is that it not only shows the hierarchy, but also the relative sizes of the files and folders. Most other ways of showing trees (like node-link diagrams) don’t show numerical data, and especially don’t reflect the way the numbers sum up when going from the branches of the tree towards the root. ## **Squaring the treemap** If the tree-map above doesn’t look very familiar, it’s probably for two reasons: it uses the original _slice-and-dice_ layout, and it’s focused on the hierarchy. The more common, modern use of treemaps uses a different layout, and usually only consists of a single level. Even when there are multiple levels, they tend to be de-emphasized. The more common treemap layout is called _squarified_, which produces the kind of visualization you probably recognize. By keeping the aspect ratio of all rectangles closer to square, categories are easier to compare, and they don’t turn into thin slivers like some of the ones in the earlier image. Below is an example of such a treemap, [the _Map of the Market_ showing the stocks in the S&P 500 index](https://finviz.com/map.ashx?t=sec). The size of each rectangle represents market capitalization for each company, and color is used to visualize the change in stock price. Note that there is a hierarchy here, with the top level defined by the sector (like Technology, Financial, Consumer Cyclical, etc.), and a level below defined by the industry (inside Technology there’s Consumer Electronics, Semiconductors, Software – Infrastructure, Software – Application, etc.). This helps both locate individual companies and show the size of each of those industries and sectors — but it can also be ignored if you just want to compare individual companies. ## **Treemaps on BI dashboards** The way treemaps are used on business intelligence dashboards today is more like the map of the market above, or the pizza treemap at the top of this post, than the original file system exploration. Treemaps are often used as an alternative to pie charts, either when there are too many items to show clearly or in [the attempt to avoid pie charts](/content/blog/truth-about-pie-charts/index.html). There’s an interesting leap here that I feel is worth pointing out. We’re used to looking at data like this as a bar chart, or some other way that doesn’t involve a hierarchy. The map of the S&P 500 index, likewise, shows a very different view of the market than the usual way of looking at stock prices, which is to show opening or closing prices [as time series](/content/blog/analyzing-time-series-data/index.html). The way treemaps are used in all these cases is to construct a hierarchy from the data, and it turns out that you can do that whenever your data contains categorical dimensions. In the examples above, it’s still fairly natural, but there’s no reason you can’t use other dimensions (like year of order date and state the pizza was ordered in, for our pizza example) to create the hierarchy you want. This is quite similar to how [cross-tabulations or contingency tables](https://en.wikipedia.org/wiki/Contingency_table) work, in fact. The difference being that cross-tabulations are not hierarchical, but they form the basis for treemaps built from tabular data. ## **The many uses of treemaps** While today’s treemaps look quite different from the original tree-map, and they’re also used differently, the underlying idea has proven to be incredibly useful. Treemaps can adapt to different sizes and aspect ratios, and allow us to show more categories than the typical pie chart, making treemaps a very effective way to visualize categorical data. Treemaps add an interesting piece of structure to a BI dashboard and can provide a useful alternative or additional view to time-series and other data. You can get started building treemaps in D3, and [explore examples of treemaps in the D3 gallery](/content/@d3/gallery#hierarchies/index.html). They’re also useful to [pack information into small spaces](/content/blog/big-insights-small-spaces/index.html), and it’s worth checking out [their relative, Mosaic plots](/content/blog/five-underused-charts-for-richer-dashboards/index.html). \]\]>https://observablehq.com/blog/avoid-radar-charts https://observablehq.com/blog/avoid-radar-chartsTue, 25 Mar 2025 00:00:00 GMTThere are many chart types to choose from when visualizing data, from the common and somewhat boring bar and line charts to more uncommon chart types like [treemaps](/content/@d3/treemap/2/index.html) and [Sankey diagrams](/content/@d3/sankey-component/index.html). Some of them (like bar charts) work for almost any kind of data, while others (like [treemaps](/content/learn/glossary#treemap/index.html) and [Sankeys](/content/learn/glossary#sankey-diagram/index.html)) can work well for specific types of data and use cases, such as hierarchies or flows. Wandering through [the data visualization zoo](https://homes.cs.washington.edu/~jheer/files/zoo/), we can also spot some lesser-known chart types that might seem like a good idea for specific kinds of data but are really never a good choice. Among this group, we find the [radar chart](/content/learn/glossary#radar-chart/index.html). In this post, we walk through what you need to know about radar charts and why you should always avoid using them when visualizing data. ## **What are radar charts?** A radar chart (or radar plot, also sometimes called spider chart because it can resemble a spider web) is a polar line chart. A number of axes are drawn out from the center, each representing a data dimension or column. For each data item, the value for each column is drawn onto those axes as a point, and the points are then connected with lines. Here’s an example of a radar chart showing the results of a 2014 survey of smartphone buyers about Nokia phones. A radar chart showing the results of a survey of smartphone buyers. Since the axes are difficult to label individually, radar charts are easier to draw when all axes share a scale. That makes them popular for surveys and personality tests, where the responses are all on the same scale, or when a percent score can be computed from the responses, such as in the phone survey example chart shown here. ## **Radar charts are often shown as multiples** Radar charts rarely come alone; they’re usually shown as multiples for comparison. Here is a version with two more phone manufacturers, and the areas are filled in. The supposed strength of these charts is that they allow comparison and show outliers. While this may be true, depending on how many and which dimensions are different from the others, it comes with some very serious caveats. ## **The problem with radar charts** While they’re seemingly straightforward and useful, radar charts have at least two big drawbacks. First, the order of data dimensions around the circle matters. One order might create a smooth and symmetrical shape, while another will produce something irregular and spiky. To demonstrate this, here is the exact same data shown four times, with only the order of the axes different between them. The second problem is that radar charts use lines to represent categorical data, which means that the lines connecting the points between the axes are inherently meaningless. If that sounds familiar, I talked about this problem in [our previous post on bar charts vs. line charts](/content/blog/bars-vs-lines-time-series-data/index.html). A radar chart is really just the polar-coordinates version of a line chart (or area chart, if we fill it in). We could show the same data as above in a linear chart: The line chart is meaningless here, and it will change shape as dramatically as the radar charts above when the order of categories changes. The bar representation, whether linear or in polar coordinates, is the better way to show this data. Plus, the bar representation doesn’t create the same visual noise. ## **Not all charts are useful for data analysis** Having so many charts to choose from is great, and it can be very entertaining to wander through the visualization zoo. But when it comes to making meaningful charts that provide information to people, we need to look at them with a more critical eye. Radar charts look interesting, and it is true that they allow people to spot outliers. But other charts do that too, and in ways that then also allow users to get meaningful information from them without visual clutter. Radar charts create more problems than they solve, and they should be avoided because they don’t convey information as effectively as [other chart types](/content/blog/five-underused-charts-for-richer-dashboards/index.html). There are many good alternatives to radar charts, from simple [bar charts](/content/learn/glossary#bar-chart/index.html) to faceted charts, and a number of other options like those you'll find in the [D3 examples](/content/@d3/gallery/index.html) and [Observable Plot gallery](/content/@observablehq/plot-gallery/index.html). \]\]>https://observablehq.com/blog/underused-options-observable-plot https://observablehq.com/blog/underused-options-observable-plotTue, 18 Mar 2025 00:00:00 GMTIn data visualization, even small design choices can significantly impact how an audience perceives, experiences, and understands the data presented. Detailed chart customization, however, can come at a cost, as developers and analysts often spend valuable time fiddling with the final design touches. And how many of us have found ourselves deep in those time-consuming customization weeds, only to find out later that a built-in option to do **just that thing** already exists? In [Observable Plot](/content/plot/index.html), there are a _lot_ of chart design options waiting patiently in documentation that make otherwise tedious changes a breeze. In this post, we shine a light on eight underused chart options in Plot. ### **1\. Curves in line charts** In most data visualization libraries (including Plot), the default way records are connected in a line chart is with a straight line from point-to-point. That’s often the best choice for [high-frequency time-series data](/content/blog/analyzing-time-series-data/index.html) collected at predictable intervals (e.g. hourly or daily data). But for sparse or uneven data where you want to visually indicate stepwise changes, or you’re just looking for a different aesthetic, you might want to change how a line is drawn between points. You can quickly update how points get connected with Plot’s [curve option](/content/plot/features/curves/index.html), which determines how values between points are interpolated to draw a continuous line. There are over twenty curve options in Plot, from _natural_ (a natural cubic spline) to _step_ (a piecewise function where _y_ changes at the midpoint of _x_). Update your chart to use any of the built-in curves by simply adding `curve: "natural"` (or another curve name) as an option within your line mark, as shown below: ``` Plot.plot({ marks: [\ Plot.lineY(numbers, {curve: "natural"}),\ Plot.dotY(numbers, {x: (d, i) => i})\ ] }) ``` ### **2\. Rounded corners in bar charts** In Plot, bar corners are square (not rounded) by default. If you want to soften the edges a bit to change the feel of your chart and better delineate between different categories (for example in a stacked bar chart), you can do so by [setting the corner radius](/content/plot/features/marks#rounded-corners/index.html). In the clip above, all corners are rounded in the same way using the `r` option — but you can even specify which corners are rounded! For example, the effect below is created using the `rx2` option, which applies a corner radius to both the x2-y1 (bottom right) and x2-y2 (top right) corners of the bar. Adding a negative `rx1` allows the bars to curve towards the ones they’re stacked on for a more seamless look, as shown below: ``` Plot.plot({ marginLeft: 150, color: {legend: true}, marks: [\ Plot.barX(\ ca_dams,\ Plot.groupY(\ { x: "count" },\ {\ y: "Primary Purpose",\ sort: { y: "x", reverse: true, limit: 5 },\ fill: "Primary Owner Type",\ rx1: -10,\ rx2: 10,\ clip: "frame"\ }\ )\ ),\ Plot.ruleX([0])\ ] }) ``` These options work with any rect-type mark, so whether you’re building bars, [waffles](https://github.com/observablehq/plot/blob/main/CHANGELOG-2024.md#0616), or heatmaps, you’ve got easy control over your corners. ### **3\. offset: “normalize”** You can use the [offset option](/content/plot/transforms/stack#stack-options/index.html) to rescale stacked data (like stacked bar or area charts). One particularly useful option is `offset: “normalize”`, which rescales stacked data to a total value of 1, allowing a viewer to more easily compare proportions across groups. Combined with `percent: true`for the scale, you get percentages in the tick marks. For example, in the chart below we use `offset: "normalize"` with percent`: true`to rescale a stacked bar chart of dam types in the United States, with color mapped to dam ownership: ``` Plot.plot({ x: {percent: true}, marginLeft: 230, color: {legend: true}, marks: [\ Plot.barX(\ ca_dams,\ Plot.groupY(\ { x: "count"},\ {\ y: "Primary Purpose",\ fill: "Primary Owner Type",\ offset: "normalize"\ }\ )\ ),\ Plot.ruleX([0])\ ] }) ``` For a different effect, stacks can also be centered on the axis using the `offset: "center`" option, or set to `offset: "wiggle"` to minimize their apparent motion around the axis. ### **4\. Faceting on a continuous quantitative variable** Usually we facet on an ordinal or categorical variable, splitting the data into small multiples based on discrete groups (like [industries](/content/@observablehq/plot-faceted-areas/index.html)) or ordinal levels (like age brackets). But you can also facet by a continuous quantitative variable. Doing so just requires an extra step to bin the values into discrete groups. Luckily, Plot’s [interval scale option](/content/plot/features/scales#scale-transforms/index.html) makes it easy to make those bins right within your Plot code (no need for outside [data wrangling](/content/learn/what-is-data-wrangling/index.html)). For example, the code below facets a subset of the [diamonds dataset](https://ggplot2.tidyverse.org/reference/diamonds.html) to visualize a histogram of diamond size (carats), faceted by price bins (using a $500 interval): ``` Plot.plot({ fy: { interval: 500, reverse: true }, marks: [\ Plot.rectY(\ smallDiamonds,\ Plot.binX({ y: "count" }, { x: "carat", fy: "price" })\ ),\ Plot.ruleY([0])\ ] }) ``` You can similarly bin continuous dates using Plot’s interval option, for example to [facet by decade](/content/@observablehq/plot-map-small-multiples/index.html). ### **5\. Add a symbol legend, get colors for free** When you add symbol and color scales to differentiate categorical groups, it’s nice to have a legend showing both. In other words, if a group is represented by a red triangle in the chart, then it should show up as a red triangle in the legend. This is one of those things that adds nice final polish to charts and [data visualizations](/content/learn/what-is-data-visualization/index.html), but is also easy to forget. In Plot, the [legend](/content/plot/features/legends/index.html) for the symbol scale automatically incorporates a color scale. That means when you set `legend: true` for the symbol scale, you get the colors for free! For example, the chart below shows how adding `symbol: {legend: true}` adds a legend with both the appropriate symbol, and the automatically integrated colors mapped to penguin species. ``` Plot.plot({ symbol: { legend: true }, color: {scheme: "Dark2"}, marks: [\ Plot.dot(penguins, {\ x: "body_mass_g",\ y: "flipper_length_mm",\ fill: "species",\ symbol: "species"\ })\ ] }) ``` ### **6\. Interactive crosshairs** In Plot, you can add instant tooltips using the [tip mark](/content/plot/marks/tip#tip-options/index.html). But did you know that there’s also an interactive [crosshair mark](/content/plot/interactions/crosshair/index.html)? Use `Plot.crosshair` to help users explore coordinates in the context of the x- and y-axis scales, as shown below for AAPL closing stock prices over time: ``` Plot.plot({ marginLeft: 50, marks: [\ Plot.line(aapl, {x: "Date", y: "Close"}),\ Plot.crosshair(aapl, {x: "Date", y: "Close"})\ ] }) ``` ### **7\. mixBlendMode** We often take for granted how colors appear when chart elements, like dots in a scatterplot, overlap. With Plot’s mixBlendMode option, you can choose from different [blend modes](https://developer.mozilla.org/en-US/docs/Web/CSS/mix-blend-mode) to control how colors blend when they overlap, for example to avoid occlusion while still keeping opaque marks. In the code below, overlapping histogram bars are exposed using different blend modes (e.g. by adding `mixBlendMode: "multiply"` within the bar mark): ``` Plot.plot({ y: { grid: true }, marks: [\ Plot.rectY(\ olympians,\ Plot.binX(\ { y2: "count" },\ { x: "weight", fill: "sex", mixBlendMode: "multiply" }\ )\ ),\ Plot.ruleY([0])\ ] }) ``` ### **8\. Spatial interpolators** That’s right, Observable Plot has spatial interpolators! In Plot’s [raster](/content/plot/marks/raster#spatial-interpolators/index.html) and [contour](/content/plot/marks/contour/index.html) marks, add the [interpolate option](/content/plot/marks/raster#spatial-interpolators/index.html) to fill a raster grid with interpolated values using the built-in nearest, barycentric, or random walk methods. Even better, they work for both quantitative and categorical values! Below, a user toggles between options to interpolate penguin species values (a categorical variable) over the raster grid: ``` Plot.plot({ width: 500, color: {legend: true}, marks: [\ Plot.raster(penguins, {\ x: "body_mass_g",\ y: "flipper_length_mm",\ fill: "species",\ interpolate: "random-walk",\ opacity: 0.5\ }),\ Plot.dot(penguins, {\ x: "body_mass_g",\ y: "flipper_length_mm",\ fill: "species",\ stroke: "white",\ r: 4\ })\ ] }) ``` ### **Keep customizing charts and visualizations with Observable Plot** We’ve shared eight useful (and underused) examples of Observable Plot’s built-in chart options, but there are [many others](/content/plot/features/marks#mark-options/index.html), including to easily add interactive [tooltips](/content/plot/marks/tip/index.html) and [pointers](/content/plot/features/interactions#pointing/index.html). Keep an eye out for upcoming [options to quickly add brushing, zooming, and animation](/content/plot/features/interactions/index.html) right in Plot! Explore the [Plot documentation](/content/plot/index.html) and [gallery](/content/@observablehq/plot-gallery/index.html) to get inspired, and to discover more ways you can quickly customize the look, feel, and behavior of charts without reinventing the wheel. \]\]>https://observablehq.com/blog/data-exploration-checklist https://observablehq.com/blog/data-exploration-checklistThu, 13 Mar 2025 00:00:00 GMTExploring a new dataset is exciting, but it’s important to understand the data before jumping in and starting to look around. It’s too easy to ignore basic data cleaning steps or make assumptions about what columns or specific values mean, or how complete the data is. Below, we list a few steps that will help you make sure that your analysis is true to the data and useful to the people who might be making decisions based on what you’ve uncovered. At the very end, we collect all the steps into a convenient checklist, so you’ll be better equipped for your next [data exploration](/content/learn/what-is-data-exploration/index.html) and analysis project. ## Before you start your data exploration: Know your data, its source and journey This first issue is often overlooked, and it leads to many problems down the road: know your data! The danger here is making assumptions about the data and not checking them, or getting data without knowing the basics such as [when it was last updated](/content/blog/align-dashboards-with-scales-of-business-decisions/index.html), whether it has been filtered, or what else has already been done to it. First, make sure you know the data’s actual source. Is this coming straight from a [database](/content/learn/glossary#database/index.html)? Which database, exactly? How was it exported (if you’re not accessing the database directly)? When was it exported? Has the data been cleaned? Even if you know the source, you should find out if the data has been processed in some way. Are you looking at a subset of the data or all of it? Has it been filtered? Have values been added to rows? Has somebody already cleaned it up (and if so, what exactly did they do)? Depending on your data and how familiar you are with it and its context, a [data dictionary](https://en.wikipedia.org/wiki/Data_dictionary) can also be crucial. It tells you what the different columns contain and what the values in each column mean. This isn’t always straightforward, because some datasets encode special values using certain numbers. 9999 might mean a missing value, or all 99xx values might mean different things. Knowing what each column contains, and what the values in it mean, are important for the next steps in data checking, but also for analysis and modeling. ## Look for inconsistencies and outliers when conducting your exploratory data analysis Once you know where your data is coming from and what has been done to it, it’s time to look at the data values themselves. This is best done by opening your data set in a data exploration platform to quickly scroll through tables, sort, and filter your data. The first thing to look for are values that stick out when just scrolling through the data. Are there any values that seem wrong, is there text in numeric columns, or anything that just looks off? Maybe values that are unusually large, or negative values where there shouldn’t be any. Next, sort by the different columns and look at the maximum and minimum values. Do they all seem reasonable? It is often difficult to set strict upper and lower limits for numerical values, so doing this by hand can be helpful. Some datasets also use specific numbers as codes for missing values, which makes this more challenging as well. Extreme and unusual values can be data errors, but they can also be a first hint of outliers to look for when you move from data exploration to the actual [data analysis](/content/learn/what-is-data-analysis/index.html). ## **Find duplicate and missing values in your dataset** The next step is a closer look at your data values to check for problems beyond individual data points. That means both missing and duplicate values. Duplicate values tend to be easier to spot by simply looking through your data when sorted by some field. You can usually notice the duplication of rows. You might have already spotted them in the previous step, in fact. Missing values are trickier, since there are two different kinds. The more obvious are individual values within rows that are empty, null, or encoded as missing. These should be easy to find if you know how missing values are encoded. Perhaps they’re obvious like empty fields or NULLs, but they might also be some special, usually large number (or -1 for values that can otherwise only be positive). Missing rows are more difficult to detect (see also the next point). If missing values inside of rows are known unknowns, these are the unknown unknowns. One way to look for them is by counting the values by some category, perhaps departments or product groups, and comparing those counts. Should they all have roughly similar numbers of records? Another way is to look for values per time period, keeping an eye out for days, weeks, or months with few or no records. ## **Watch out for incomplete time periods before analyzing data** Perhaps the most insidious problem in data analysis are incomplete time periods at the end of a dataset. If you’re looking at your data by month, for example, the current month might look worse than previous ones. But is that because your sales were lower, you had fewer clicks, etc., or simply because you don’t have a full month’s worth of data? The difficulty with this is not only that the current period is usually the most interesting, but also that you might not know what granularity will be of interest to your users. On top of that, time periods don’t always line up in a convenient way, especially weeks and months. Here, it becomes an issue of knowing your users and what they’re looking for. If the current time period has to be included, make yourself a note to point out that it’s incomplete, and present your findings in that context. If you can, it is usually better to exclude incomplete time periods to avoid confusion. However, in the reality of data analysis this is rarely possible or desirable. ## **A checklist for navigating and exploring data** Here are the above steps for checking your data again, in convenient checklist form: - Where does the data come from? - When was it exported (if not directly accessed from a database)? - Has it been filtered? - Has it been cleaned? - Have computed fields been added? - Do you have the data dictionary with column definitions and special values? - Have you checked for extreme values and outliers? - Have you checked for duplicate values? - Have you checked for missing values within rows? - Have you checked for missing records? - Do you know what incomplete time periods there are at the beginning and end of your dataset? Checking your data is not the most exciting part of data analysis, but it is a crucial step. Any analysis you do on data you haven’t carefully checked might end up being invalid or misleading the people you do it for. By following these steps to sanity check your data, you’ll be better prepared to explore your data, conduct a thorough analysis, and build data visualizations. Get started with exploring data and creating visualizations with [Observable Canvases](/content/platform/canvases/index.html), our new collaborative data whiteboard. Observable Canvases brings stakeholders and data teams together to facilitate data exploration and analysis, and unlock insights faster. \]\]>https://observablehq.com/blog/bars-vs-lines-time-series-data https://observablehq.com/blog/bars-vs-lines-time-series-dataThu, 06 Mar 2025 00:00:00 GMTRecently, we looked at [how bar charts work](/content/blog/simplicity-bar-charts/index.html), why they work well for categorical data, and what histograms are useful for. We also briefly touched on line charts, and why they aren’t a good choice for categorical data. And yet, while they are vastly different, they are sometimes used for the same data: time series. In this post, we dig into that “why” a little more closely, compare bar and line charts directly, and discuss when bar charts or line charts are the right choice for temporal data. ## **Bar charts vs. line charts** Bar charts are commonly used for categorical domains such as products, countries, or languages. These are distinct values that can be reordered based on a value (such as population or GDP, in the case of countries). They’re also discrete values, with nothing between them. Line charts, on the other hand, work well for continuous domains, especially time. Of course bar charts are used for time too, and we’ll get to that in the next section. But the opposite doesn’t work: lines can’t be used for categorical data. Here is an example dataset of letter frequencies in English  (which you may remember from the previous post), shown as a bar chart and also as an ill-advised line chart. This doesn’t look terrible as a line chart. Since the values are sorted by frequency, we get a smooth line. But let’s see what happens when we reorder the values: These jagged lines imply a dramatic change, when all we did was reorder the points. The bar chart looks different too, of course, but we can still read the individual bars, and figure out which bar corresponds to which letter. The line chart always creates a continuous shape, which makes no sense on a categorical axis because it implies that you can interpolate between the discrete data values. Its jagged shape means nothing, because only the points actually exist as values, the lines between them are completely arbitrary. This example illustrates why using line charts for categorical data is a really bad idea, and many applications and libraries make it difficult to do for a reason. The line chart is mostly visual noise, is hard to read, and doesn’t fit the data type. A bar chart is a much better choice. ## **When to use bar charts for time series data** Time and time series data are a little more complicated. Technically, a chart showing data over time should use a line chart, since time is continuous — except when it isn’t. When we look at many data points over time, a line chart is the obvious choice. Consider this example of [Apple’s closing stock price](https://finance.yahoo.com/quote/AAPL/) over several years. This chart shows every single business day over more than four years, which is a lot of records (1,260 data points, to be precise). While this isn’t continuous data per se (the closing price doesn’t smoothly vary over time, there’s only one per day), when the number of data points is so large relative to the space they’re shown in, they might as well be continuous. If we were to aggregate this data, however, the chart would look different. Here I’ve computed the average closing price of the stock for each year. This is much less interesting to look at. But more importantly, we’re no longer looking at truly continuous data since we have reduced our data to only one value (mean closing price) per year. The bar representation seems a lot more apt here, and also makes the chart easier to read. Since there’s now only one data point per year, the notion of continuous change no longer applies. Natural phenomena, such as temperature or a person’s weight over time, change in a continuous way. When the temperature is 60 degrees Fahrenheit at 7am and 65 degrees at 8am, [it passes through all the values in between these two measurements](https://en.wikipedia.org/wiki/Intermediate_value_theorem). We’ve only observed these two data points, but we know that the temperature existed at all times between the two points, and that it passed through 62 degrees, 63 degrees, and 64.879331 degrees at some point. That is not the case for a company’s total sales month-to-month, or the yearly average of a stock price. When aggregating data over time, the resulting data is no longer smooth and continuous, and so it makes sense to show the values as bars instead of lines. This is sometimes done for aesthetic reasons as well, because it improves legibility for charts when there are only a few values. ## **When to use stacked bar charts** While bars can be read and compared with great precision when they’re next to each other, this isn’t the case when they’re stacked. I touched on this in [our blog post on pie charts](/content/blog/truth-about-pie-charts/index.html), since pie charts are really more comparable to stacked bar charts than regular ones. Consider [this stacked bar chart of population by age group in all U.S. states and territories](/content/@d3/stacked-bar-chart/2/index.html). It’s a perfectly fine bar chart if you’re looking at the total population in each state. But if you wanted to compare a specific age group, like say the 40-49-year-olds in yellow, things would get trickier. The yellow bars don’t sit on a common baseline, so they’re much harder to compare. Also, finding out the fraction of this age group of the total population wouldn’t be any more accurate than if it were shown in a pie chart (and probably a lot less precise in the shorter bars). That stacked bar charts are hard to read has been known for a while, in fact [a famous study published by Cleveland and McGill in 1984](https://thork.people.uic.edu/fair/cleveland&mcgill_jasa84.pdf) already reported this. It was later expanded upon in a [paper of bar chart experiments in 2014](https://www.tableau.com/research/publications/four-experiments-perception-bar-charts). Stacked bar charts are popular because they show more data in the same space. But this can also overload them with information (and color!), and much of the added information is very difficult to read. There are better options for visualizing and comparing subsets of datasets, such as [faceted plots](/content/@observablehq/plot-faceted-areas/index.html) that put each bar chart on its own base line — or, if the focus is on part to whole comparisons, even pie charts! ## **Building a line series chart with Observable Plot** You can quickly build a time series line chart with a few short lines of code using Observable Plot. Simply copy the below example snippet into a JavaScript cell in an Observable Notebook, or select “Line chart” from the searchable new cell menu. Connect your data, and update the data and variable names to match what’s in your data. Plot will automatically handle scales, axes, and tooltips, helping you quickly generate a time series line chart: ``` Plot.plot({ marks: [\ Plot.ruleY([0]),\ Plot.lineY(aapl, { x: "Date", y: "Close", tip: true })\ ] }) ``` ## **Conclusion: Bar charts are usually the best fit for categorical data** Beyond the obvious use for categorical data and histograms, bar charts are sometimes the right option, and other times there are better ways to visualize data. While it would be nice to think that they’re always the better choice, it’s often a matter of taste and judgment where they work best. It’s even possible to simply overload them with information, such as in many stacked bar charts. Where bar charts are the undisputed king, however, is when showing categorical data. The way we can easily and accurately read and compare them, and the fact that they can be ordered in different ways, makes them the obvious (and really only) choice in that case. That one use case aside, there are always nuances and judgment calls. Knowing the rules, and when to break or ignore them, is what makes visualization so fascinating. [Learn more about bar charts](/content/blog/simplicity-bar-charts/index.html) from our previous post on the topic, and explore more chart types we’ve covered on this blog, like [pie charts](/content/blog/truth-about-pie-charts/index.html)! \]\]>https://observablehq.com/blog/maps-and-data-visualization-with-fil-riviere https://observablehq.com/blog/maps-and-data-visualization-with-fil-riviereMon, 24 Feb 2025 00:00:00 GMTStart poking around Observable — on the platform, in the codebase, in user support, across our data visualization galleries — and you’ll find Philippe Rivière’s fingerprints _everywhere_. Fil (as we know him) is a Data Visualization Engineer at Observable, and a core contributor to our open-source software including Observable [Plot](/content/plot/index.html), [D3](https://d3js.org/), and Observable [Framework](/content/framework/index.html). His avatar (cropped from an [illustration by Mad Meg](https://madmeg.org/lenragee/#5.288/0.341/0.808), which Fil describes as “a character sitting in a corner watching the horror of the world with a lot of angst”), appears atop many of our most popular chart examples and alongside countless responses in notebooks and discussion boards: Fil has had an especially outsized impact when it comes to helping people to make better maps, including as the primary developer of Plot’s [geo mark](/content/plot/marks/geo/index.html). So it wasn’t a surprise when I reached out to Fil about doing an interview that his response was simply: “We can always talk about maps.” Read on to hear, in Fil’s own words, about his path into [data visualization](/content/learn/what-is-data-visualization/index.html), what he likes in a map, what he’s excited about next, and why — despite developing tools that help us all make maps faster — he’s wondering if we need a “slow cartography movement.” ## **Q: How did you get into data visualization?** **A:** I used to be a journalist. When I was just getting started in journalism, I was responsible for building and maintaining the paper’s website. I worked at that same newspaper for almost twenty years. I eventually got fed up and decided I wanted to try something new. At the same time, my friend and colleague [Philippe Rekacewicz](https://www.visionscarto.net/_philippe-rekacewicz_) (who was a cartographer at the same newspaper) had decided to leave. I loved what Reka — as everyone calls him — was doing with maps, so I started learning cartography with him. Reka likes to draw maps by hand. He goes through datasets line-by-line, checks everything, makes sense of what it means, then draws a map with pastel crayons. This seemed to me like a tedious process, because I was used to programming. So, I started learning from Reka while trying to contribute to the work. I wanted to see how we could automate making maps for the web. That’s really how I stumbled onto [D3](https://d3js.org/). Finding D3 was so exciting. I started off making a lot of data visualizations and freelanced as a data visualization developer for six or seven years. Now I work full-time at Observable. So, I guess my path into data visualization was through discovering something that was exciting to me and trying to learn it. ## **Q: What about maps do you find exciting or impactful?** **A:** I like seeing the relationships between things — the movements. A map that describes trade, for example, is more exciting than a map that shows places. I don’t really care for GIS ( [Geographic Information Systems](https://en.wikipedia.org/wiki/Geographic_information_system)). I’m not interested in placing things precisely on a map. I’m way more interested in drawing networks and flows and differences between one region and another than I am in marking locations. Several data visualizations that I find really striking are in [William Bunge](https://en.wikipedia.org/wiki/William_Bunge)’s Nuclear War Atlas, which is a collection of maps about nuclear armaments. One of the maps that I find most impressive shows how a radioactive plume from a nuclear explosion travels day-by-day. You can see that any country that drops a nuclear bomb anywhere in the northern hemisphere is basically just nuking itself. The path of a radioactive plume, circumnavigating the world and reaching the original explosion site in just over one week, from Nuclear War Atlas by William Bunge. Here’s another map from the same book, showing the national boundaries between the Soviet Union and the United States if considered by their nuclear power and reach (in 1988, when the book was published). Basically, it shows that the U.S. nuclear weapons are everywhere and can reach any place on the planet. And, the Soviet Union has the same power. Basically, the national boundaries have dissolved. From Nuclear War Atlas by William Bunge. These are maps that have a story, have a purpose, and are very striking. For me, these are the types of maps that really amaze me and teach me something. As another example, I worked on maps for Sonia Shah’s book [The Next Great Migration: The Beauty and Terror of Life on the Move](https://soniashah.com/thenextgreatmigration/). Here’s one showing how the Pacific was populated in prehistoric times. I like making these types of maps because they are evocative. They try to distill the current knowledge about how people have moved. A map of human dispersion across the globe, by visionscarto (2020), for Sonia Shah. Read more in Sonia’s article [Migration is the earth’s connective tissue](https://www.visionscarto.net/next-great-migration). This map features the first published use case of IMAGO, a projection [recently added in D3](https://github.com/d3/d3-geo-polygon?tab=readme-ov-file#geoImago) by Fil and collaborators. In some cases, you have to share things that aren’t meta-geographic and that do require GIS. But there is a continuum between GIS, which focuses on precisely describing where things are, and these types of maps that help to explain how or why something happened. In between, there are a lot of stories you can tell. With Reka and other friends, we created a collective called [visionscarto](https://www.visionscarto.net/en) where we share articles, usually related to maps and representation. I like the types of maps we tend to share in visionscarto because their authors are careful not only about what they describe, but also about how they describe it. ## **Q: Can you describe some cartography projects you’re particularly proud of?** **A:** One of the first things I wanted to do for Reka was to automate the Bertin projection, which is used a lot by the French school of cartography. It’s a nice projection because it preserves almost all of earth’s surfaces without distorting continents too terribly, with more distortion pushed to the oceans. It was created by Jacques Bertin in 1953, but (in 2013) there was still no formula to describe it. It only existed as a drawing, with people passing around copies of a base map in Adobe FreeHand. When I was first starting this project, I contacted Anne Le Fur (a student of Bertin) who explained to me how Bertin had drawn the projection by hand, with a globe and a light source. Then, I started to study D3 in depth to understand the incredible [d3-geo](https://d3js.org/d3-geo) module that Mike Bostock and Jason Davies had created a few years before. I thought that automating the Bertin projection by computer code would be a weekend project. It took me around 4 years. Finally, we were able to describe the projection mathematically and [implement it in D3](https://github.com/d3/d3-geo-projection/blob/main/src/bertin.js) in 2017. A map of earth made with the Bertin projection from [visionscarto](https://www.visionscarto.net/earth-bertin-projection). Learn more about [Fil's work to automate the Bertin projection](https://www.visionscarto.net/bertin-projection-1953). Another project I really loved was the [30-day map challenge](https://30daymapchallenge.com/) of November 2021, because every day I was playing with [a new approach or algorithm](/content/@visionscarto/loir-et-cher-isodemographique?collection=@visionscarto/30daymapchallenge/index.html). Some were things that had been on my mind for a long time, and I wanted to publish something about each of them in a single day. Others I really did just develop during that month. I did not sleep much. One of the 30-day map challenge pieces I’m very proud of is an isodemographic map of where I live. In this map, each cell represents the same number of inhabitants. Fil’s isodemographic map, where each cell represents the same number of people. Visit [the notebook](/content/@visionscarto/loir-et-cher-isodemographique?collection=@visionscarto/30daymapchallenge/index.html) to explore other metrics, and to see the code. Usually the way you do this is by deforming shapes using a tool such as [ScapeToad](https://github.com/christiankaiser/ScapeToad) (a Java app that grows or shrinks regions so the area matches the size you want to represent). If you look at those maps, they’re based on a grid that gets morphed, and the result is visually unappealing. The concept however is nice: you want to present something that is proportional to the population, or the economy, or whatever. This map uses a different algorithm called sliced optimal transport, which I learned about the preceding year in a [workshop](https://transpopt-igrv.sciencesconf.org/index.html) organized by French researchers, including Julie Delon and David Coeurjolly. I learned a bit about the theory, the tools, and the algorithm. The image it produces is also organic, but very nice-looking. I don’t think the method had ever been used in cartography before. I don’t know why I didn’t make a whole atlas of these. _(Explore_ [_Fil’s entire 30-day map challenge collection_](/content/collection/@visionscarto/30daymapchallenge/index.html) _on Observable.)_ ## **Q: What’s the hardest part about making maps and data visualizations?** **A:** Everything is hard. That’s the thing with [data visualization](/content/learn/what-is-data-visualization/index.html). You have to understand the data. You have to understand the context. You have to understand the technique. You have to be an artist. You have to understand composition and color theory. All of these are really hard in a practical way, and in a theoretical way. Often you see a map and say “Oh it’s nice, I can do that.” An old boss of mine used to look at Reka’s maps and say, “This is a drawing. My nephew could do that.” But everything about maps and data visualization is hard. ## **Q: What’s a “hidden gem” in Observable Plot?** **A:** I’m very proud of [spatial interpolators](/content/plot/marks/raster#spatial-interpolators/index.html) in Plot. Especially the [random walk interpolator](https://github.com/observablehq/plot/blob/7c48dac18ccdb927a9d7794d95180ffd467dfb49/src/marks/raster.js#L432). The [walk-on-spheres technique](/content/@fil/walk-on-spheres/index.html) is so beautiful. It was made popular recently by Rohan Sawhney and Keenan Crane, who showed how it can approximate computations that are hard to do otherwise. It’s a good [Monte Carlo approach](https://www.cs.cmu.edu/~kmcrane/Projects/MonteCarloGeometryProcessing/index.html) to interpolation problems. And it’s fast! A map of the strength of earth’s magnetic field, created with [Observable Plot’s raster mark using the built-in random walk interpolator](/content/plot/marks/raster/index.html). You can even mix or blend categorical values, which might be surprising. Usually, when people think about interpolation, they think of something like taking an average of numeric values, weighted by distance, to interpolate a value for a new location. The beautiful thing is that the random walk interpolator also works for categorical data. And you can blur them afterwards. For example, here we have the [iris dataset](https://en.wikipedia.org/wiki/Iris_flower_data_set), and we have three different species. It wouldn’t make sense to do a quantitative interpolation (the “average species”). But you can use random walk to select a species, and that gives you a good idea of the zone of influence of each species, and the noise in the image is also helpful to visualize uncertainty. I think spatial interpolation in Plot is a gem, in a sense, because it’s not available in any other graphic library that I know of. The random walk interpolator, implemented in Observable Plot’s raster and contour marks, applied to a categorical variable (iris species) with variable blur. Explore the random walk, nearest (Voronoi), and barycentric interpolators in Fil’s [Random walk dithering notebook](/content/@observablehq/random-walk-dithering/index.html). _(To learn more, read this short paper by Fil and Mike Bostock:_ [_Spatial interpolators in Plot_](https://journee-visu.github.io/2023/submissions/Riviere_Bostock.pdf) _.)_ ## **Q: What’s something you’re excited about?** **A:** This is something I’ve been excited about for the last decade: using the cartographic metaphor to make a map of something that is not geographic by nature. Basically, being able to make a map of anything, and leveraging people’s intuitions (of how to read a map) to make the data easier to understand. A great example is making a map that visualizes patterns in a bunch of text. This is something [Ian Johnson](https://enjalot.github.io/) is working on with [Latent Interfaces](https://latentui.com/). With sentence transformers and embeddings and so on, you can project any type of data to a space where things that are similar or have a similar meaning are closer than things that are different. [Ian Johnson](https://enjalot.github.io/)’s [clustering and visualization of ~50k U.S. federal laws](https://enjalot.github.io/latent-scope/us-federal-laws), based on titles, using [Latent Scope](https://enjalot.github.io/latent-scope/). It’s the complement to Waldo Tobler’s [first law of geography](https://en.wikipedia.org/wiki/Tobler%27s_first_law_of_geography#:~:text=The%20First%20Law%20of%20Geography,specifically%20for%20the%20inverse%20distance): things that are closer in space have similar characteristics than things further apart in space. You can take it very literally and do mathematical things like spatial autocorrelation. But, you can also do this very metaphorically and invert the proposition: if two things are similar, put them in the same neighborhood, and if they’re very very similar, on the same street. If they’re somewhat similar, then they are in the same city, and if they are very different they are in different countries. Then, you can name the clusters and show connections between places. For example, if the text is about scientific fields, you might have clusters for biology to chemistry and you could see the subjects along the way connecting them. You could apply it to knowledge, any type of textual data, even art — there’s no real limit. And while the idea of mapping non-geographic data isn’t new (indeed, we can trace it all the way back to [Carte du Tendre](https://en.wikipedia.org/wiki/Map_of_Tendre)), it is still exciting to think about new approaches and applications. We have algorithms that you can apply to abstract points, and still get something that looks like a map. For example, I worked on a project with researcher Dario Rodighiero to [visualize a network of 8,000 student theses](https://dariorodighiero.com/Super-Vision) from the École Polytechnique Fédérale de Lausanne, based on a lexical distance. This map uses the [Urquhart graph](https://en.wikipedia.org/wiki/Urquhart_graph), which evokes a road network very well, even though it’s purely a topological computation. A network visualization of student theses from École Polytechnique Fédérale de Lausanne to explore similarities and trends over time, using the Urquhart graph (source: [dariorodighiero.com](https://dariorodighiero.com/Super-Vision)). See an example of the [Urquhart graph implemented in an Observable notebook](/content/@fil/world-cities-urquhart/index.html). A map of non-geographic data doesn’t necessarily make more sense than any other abstract way of seeing the data, but it makes sense to use the cartographic metaphor since people already know how to read maps. We need this familiarity, otherwise all these charts are just too hard to grasp. ## **Q: How has your perspective on making maps evolved?** **A:** When I started making maps, I thought that automation would be a good idea. I wanted to make a “machine” that would save Reka a lot of time. I’ve since realized that there is a lot of value in making maps slowly. When you’re making a map, there are all these challenges happening at the same time. The technical challenges, the cognitive challenge of understanding what the data represents, using the right form, the right colors, the right projection, and so on. Strangely, it feels like older maps were better at solving these challenges, probably because they required so much manual effort and time. People would spend months making a single map. They had time to think about it. Now, we want to make maps quickly, and we have tools that are imposing their form. That means we’re assembling maps from existing parts, but the parts are more rigid. Newer maps are obviously richer in some ways, because you can add things like zooming and brushing that you can’t just do with paper and pencils and ink. But, at the same time, the shorter production time means you spend less time with the data, and that means you might oversee or exclude something important. For example, sometimes I see maps using a dataset as the source of truth. If a point is missing in the dataset, then it’s often excluded from the map, and people will just say, “Oh, there’s no data.” But that’s not true. It’s not that there’s no data, there’s just no data in this dataset. Take a statistical indicator of literacy by country. That value is likely not changing much from year to year for any given country, so we may be able to infer literacy data from another recent year. But, we tend to not do that research anymore. We don’t say “Oh, that’s an interesting question, I have to call someone to find the answer,” or try to interpolate or extrapolate values from existing data. We probably need a slow cartography movement, similar to the [slow food movement](https://en.wikipedia.org/wiki/Slow_Food). And there are still people who are doing the work very slowly, even though it’s much easier to make quick maps that have gaps. Taking the time to go through everything carefully, and asking questions when something isn’t obvious, is really important. A hand-drawn map by Philippe Rekacewicz (using the Bertin projection) shows routes of rhino horn trade, with data from the Oxpeckers Center for Investigative Environmental Journalists. Source: [Routes of rhino horn](https://www.visionscarto.net/routes-of-rhino-horn) by Fiona Macleod, on visionscarto. * * * _Fil Rivière is a Data Visualization Engineer at Observable. See more of his work on_ [_Observable_](/content/user/@fil/index.html) _, and at_ [_visionscarto.net_](http://visionscarto.net/) _._ \]\]>https://observablehq.com/blog/agropatterns-observable-greenhouse-analytics https://observablehq.com/blog/agropatterns-observable-greenhouse-analyticsTue, 11 Feb 2025 00:00:00 GMTIn Colombia’s high-altitude greenhouses where flowers and crops require careful cultivation, farmers are turning to data to stay ahead. With shifting climate patterns and increasing pressures on agriculture, the digital transformation of greenhouses is becoming essential for improving yields and protecting crops. Leading this transformation is Gabriel Coch, a veteran in mapping and geospatial technologies, with [AgroPatterns](https://www.infopatterns.com/) — a data-driven platform that’s improving how farmers monitor crop health, mitigate pests, and optimize productivity. Gabriel Coch, Founder of AgroPatterns Gabriel’s journey into agritech follows decades of experience in digital mapping, location-sharing applications, work in emergency management, and even collaborations with industry pioneers like Ray Ozzie, the creator of Lotus notes. But his work with AgroPatterns has become his most personal and impactful endeavor. ## The problem: a data void in horticulture In the cutthroat world of fresh flowers — a $30 billion industry — timing is everything. Growers must anticipate market peaks for events like Valentine’s Day and Mother’s Day, ensuring their products bloom precisely when demand surges. To meet those demands, farmers have long relied on intuition, and local and historical knowledge. But as climate change introduces new unpredictability, traditional cultivation methods can benefit from real-time data and predictive analytics. Temperature fluctuations, pest infestations, and disease outbreaks can jeopardize entire harvests, and past monitoring methods (which still often rely on paper logs and Excel spreadsheets for many farmers), can be inefficient and slow to reveal critical insights. In an industry where a single pest outbreak can wipe out millions of dollars in inventory within 48 hours, waiting for data to be compiled and analyzed can be a very costly delay. ## Digitizing the greenhouse AgroPatterns gets farmers deeper insights about their crops by integrating traditional manual record keeping with real-time sensor data, predictive analytics, and digital mapping. The core of the platform relies on a structured data collection method designed specifically for greenhouse environments, where every plating row and section is coded for consistency and traceability. A network of field scouts traverse greenhouses weekly, manually identifying pests and diseases. Instead of scribbling notes on paper, scouts input observations into a simple offline-compatible app, syncing data instantly to the cloud when the device reconnects to signal to transmit its offline data. Each greenhouse is divided into structured zones, assigned unique identifiers rather than relying on GPS coordinates (which drain mobile batteries too quickly in the field). This approach ensures that every data point is linked to a physical location within the greenhouse, creating a reliable dataset that informs decisions. “We are able to identify problems as quickly as possible,” Gabriel explains. “Catching mites in the egg stage is exponentially cheaper and less toxic than dealing with an outbreak of adults. The goal is to reduce blanket spraying and instead limit spraying to targeted zones, which saves money and reduces chemical exposure for workers and consumers alike.” Observable plays a key role in processing and visualizing the collected data. Gabriel has built a suite of [Observable Notebooks](/content/platform/notebooks/index.html), leveraging [D3](https://d3js.org/) and [Observable Plot](/content/plot/index.html) for geospatial mapping and using [Observable Framework](/content/platform/framework/index.html) to transform raw field data into actionable visual insights. An Observable Notebook that Gabriel created to procedurally generate greenhouse maps. Using geospatial overlays built with Observable and Mapbox, farmers can view real-time maps that display the status of pest infestations, chemical treatments, and productivity metrics at a glance. These overlays replace the previous digitization process, where each field’s layout of latitude and longitude points had to be painstakingly mapped and updated separately. Now, Gabriel’s Observable Notebook automates the entire process — inputting just a few key reference points generates a complete geospatial representation of the greenhouse down to the individual planting beds. ## From the fields to the cloud: AI and automation AgroPatterns uses AI-driven automation to make data more accessible for farmers. Its WhatsApp integration delivers real-time alerts and insights directly to field workers, ensuring quick decision making without reliance on in-office dashboards. Thousands of messages are exchanged daily, making this chat-based system the primary interface for field scouts. AgroPatterns' WhatsApp integration keeps farmers informed on the go. Workers can log updates using voice commands with AI-powered transcription, eliminating the need for spreadsheets or handwritten notes that can be lost or misrecorded. Farmers can also request AI-generated insights — such as identifying pest hotspots — and receive an instant geospatial map highlighting trouble areas. “People in the field aren’t sitting in front of dashboards all day,” Gabriel notes. “They need immediate, actionable information.” AgroPatterns' growing degree days dashboard, built with Observable Framework, predicts crop growth for optimal harvest timing. One of AgroPatterns’ most powerful tools is its use of growing degree days — a metric that tracks accumulated heat over time to predict plant growth stages. By continuously collecting temperature data via custom-built sensors, AgroPatterns provides farmers with precise insights on when their crops will be ready for market. “Temperature determines how fast plants grow,” Coch says. “If you’re two days off in predicting flower maturity, that’s the difference between a perfect Valentine’s bouquet and a wasted shipment.” This level of precision allows farmers to fine-tune their harvesting schedules, ensuring that every stem is cut at peak readiness. Combined with AI-assisted analysis and WhatsApp alerts, the system helps growers anticipate issues before they become costly crises. ## Scaling the impact: a vision for the future AgroPatterns currently serves over 60 farms, covering approximately 15% of the Colombian market. But Gabriel sees a much broader application for this technology. While his current focus is on flowers, he believes the same principles can be applied to other high-value perishable crops. “This is about improving agricultural practices,” he asserts. “Many small farmers lack technical expertise, but with simple tools, we can help them make better decisions, reduce chemical use, and increase their profitability.” As AgroPatterns continues to evolve, Gabriel remains a staunch advocate for Observable’s role in his workflow. His ability to iterate quickly, prototype maps, and deploy solutions with minimal overhead has been a game-changer. But beyond technology, what drives him most is the human impact. “Constant exposure to chemicals has many detrimental health effects on those that work in the field. If we can reduce unnecessary spraying, we’re not just saving money — we’re saving lives.” For Gabriel Coch, this is more than a business. It’s a mission to make agriculture smarter, safer, and more sustainable — one data point at a time. \]\]>https://observablehq.com/blog/simplicity-bar-charts https://observablehq.com/blog/simplicity-bar-chartsTue, 04 Feb 2025 00:00:00 GMTBar charts, [unlike pie charts](/content/blog/truth-about-pie-charts/index.html), are the uncontroversial workhorse of [data visualization](/content/learn/what-is-data-visualization/index.html). They fit many different kinds of data, and are the easiest to interpret when you need to read small differences in numbers with a high degree of precision. In this post, we’ll look at how bar charts work and a few typical uses. In a follow-up post, we’ll then go a bit further into more unusual use cases. ## **Categorical data: the basic bar chart** At its most basic, a [bar chart](/content/learn/glossary#bar-chart/index.html) shows a numeric value broken down by a category. This might be sales broken down by department, average yearly cheese consumption by age group, or the number of sunny days per year for different cities. Note that there are two different sources for the bar length here: it can be a numeric column in the data set (like sales or cheese consumption) or based on the number of records with a particular value (number of days tagged with “sunny”). This number determines the length of the bars. The other (usually horizontal) axis is categorical, which means that there is no inherent ordering – or, if there is an inherent order, it’s less important than the numbers shown by the bars. The categorical axis just breaks the data down into individual subsets that we care about. Take this distribution of letter frequencies in an English-language corpus, for example: If we want to know the most frequent letters, sorting by frequency (on the vertical axis) makes the most sense. We can easily find the most and least common letters, the third-common, etc. But what if we’re interested in a particular letter? Or if we want to see if there are any obvious dependencies between neighboring letters in the alphabet? Then we can use this chart, with letters arranged alphabetically (instead of ordered by count): Bars are independent from each other, so no matter how we order them, they always look the same. This is very different from a line chart, where the slope between two points can change dramatically depending on how far apart they are, and how the values differ. The fact that we can do this with bar charts is important, because it allows us to reorder bars in whatever arrangement we think will be most useful to the viewer. This is not something we would be able to do if the horizontal axis were time, for example (but stay tuned for the next post that will discuss bars on time axes). Building bar charts is straightforward in most data visualization tools and libraries. For example, here’s how to do it in [Observable Plot](/content/plot/index.html), which makes the mappings from data to visual features easy to see. The bar mark contains the dataset, _alphabet_, which has two columns here: letter and frequency. We map one to the x axis (position) and one to y (length). ``` Plot.plot({ marks: [\ Plot.barY(alphabet, {\ x: "letter",\ y: "frequency"\ }),\ Plot.ruleY([0])\ ] }) ``` If we want to sort the bars by their length, we can insert a [sort option](/content/plot/features/scales#sort-mark-option/index.html) into the bar definition: ``` sort: {x: "y", reverse: true} ``` ## **Comparison: two-sided bar charts** Technically, bar charts with vertical bars are called _column charts_, but people who insist on this distinction tend to not be much fun at parties. Bars can be horizontal too though, like in this example showing population growth for each state in the U.S. between 2010 and 2019: It’s really just another bar chart, but the horizontal bars lend themselves to annotation (names of the states and percent of change), and the vertical line reinforces the difference between states that decreased or increased in population. This is brought out even more by the color, which is magenta for negative values, and green for positive. The entire definition of this chart using Observable Plot is a little involved ( [you can see it here](/content/@observablehq/plot-state-population-change/index.html)), but the key part is just this [barX](/content/plot/marks/bar#barX/index.html) mark (the X indicating it is a horizontal bar chart). The fill property contains a function in this case, which changes the color of the bar depending on whether its value is positive or negative. The bars are also sorted from smaller to larger numbers, which puts the largest decrease at the top, and the largest increase at the bottom. ``` marks: [\ Plot.barX(statepop, {\ x: "value",\ y: "State",\ fill: (d) => d.value > 0,\ sort: { y: "x" }\ }),\ ```\ \ Despite its simplicity, a bar chart can be dressed up to be more interesting to explore, and also more informative.\ \ ## **Histograms: the one-dimensional bar chart**\ \ I mentioned earlier that a bar’s length can be based on how often a category is found in the data. But what if the data isn’t even categorical? Then we can _bin_ the values into ranges, and count those to make a [histogram](/content/blog/designing-better-histograms/index.html).\ \ What is a [histogram](/content/learn/glossary#histogram/index.html)? A histogram is a type of bar chart that shows the distribution of a set of continuous numerical data, where data is grouped into _bins_ that display how many data points fall into each bin. They are both extremely simple and useful, but also unfamiliar enough to be confusing to many people.\ \ \ \ What makes them simple is that they’re really just one-dimensional: one axis is a value, while the other is created from the dataset itself by counting how many values fall into each range of values. Since the value is usually continuous, what is counted is not individual values, which might all be different, but how often values occur within each range, or bin.\ \ ```\ Plot.plot({\ marks: [\ Plot.rectY(olympians, Plot.binX({y: "count"}, {x: "weight"})),\ Plot.ruleY([0])\ ]\ })\ ```\ \ Take a look at this example of the weights of Olympic athletes, for example. The bins here are 2kg wide. The height of each bar is how many athletes have a recorded weight within that bin.\ \ Here’s the Observable Plot code that created this chart. Note the [binX transform](/content/plot/transforms/bin/index.html) that performs the binning, which both quantizes the values for the x axis, and counts them for the y. The result then goes into the [rectY](/content/plot/marks/rect#rectY/index.html) mark to create the bars.\ \ Histograms are useful to judge the distribution of values, which means they’re read more as a shape than individual bars. In this case, we can see a fairly typical normal distribution with an interesting skew to the left (lower values). The general population would look more symmetrical, and also presumably shifted to the right a fair bit. This dataset contains weights for top-of-their-game athletes, most of whom are probably competing in sports where being lighter is an advantage.\ \ One important issue with histograms is that the bin size can dramatically change their appearance. In the example above, we didn’t define the bin width ourselves, so Plot chose a bin size of 2kg. If we instead specify 1kg or 5kg, we get a spikier or smoother histogram.\ \ \ \ What’s more, shifting the starting point of the bins left or right can change the histogram too, especially when there are many bins. It is therefore important not to read too much into individual bars, and usually better to use a larger bin size. A common recommendation for picking the right bin size is [Scott’s Rule](https://en.wikipedia.org/wiki/Scott%27s_rule), which is used in many statistical and chart libraries (including Plot) to pick bin sizes by default.\ \ Histograms are a special type of bar chart where we can’t just change the order (since it depends on the value on the axis). They’re useful to see how values are distributed, what the shape of the distribution is, and if there are outliers (especially unknown and null values).\ \ ## **Building a bar chart with Observable Plot**\ \ You can easily build a bar chart with Observable. Check out the below example code for visualizing data with a bar chart using Observable Plot. Simply copy the code into a new Observable Notebook or select “bar chart” from the new cell menu, then update the data and variable name to visualize your data sorted by y-value in descending order.\ \ ```\ Plot.plot({\ marks: [\ Plot.barY(dataset_name, {\ x: "categorical_var",\ y: "quantitative_var",\ sort: { x: "y", reverse: true }\ })\ ]\ })\ ```\ \ ## **Conclusion**\ \ Charts don't have to be fancy to be useful, the humble bar chart is perhaps the best illustration of that. Despite its simplicity, it is useful in many contexts, whether showing data values directly, aggregated numbers, or counts in a histogram.\ \ Bar charts and histograms hit a sweet spot of working well for the kinds of data that happens to often be of interest in [data analysis](/content/learn/what-is-data-analysis/index.html) and [data exploration](/content/learn/what-is-data-exploration/index.html). While there are many other chart types, none can match the ubiquity and usefulness of bar charts.\ \ Discover more tips and tricks to building effective data visualizations on the Observable [blog](/content/blog/category/data-visualization/index.html).\ \ \]\]>https://observablehq.com/blog/linked-brushing https://observablehq.com/blog/linked-brushingTue, 28 Jan 2025 00:00:00 GMTReal-world systems are complex, with patterns often influenced by a large number of intertwined variables that may be correlated. When visualizing such data, it's common to build a suite of simpler charts each showing relationships between two variables at a time. While that has the benefit of making each individual chart more manageable to build and understand, it can also mean sacrificing insights into the interconnectedness that links data together across variables in different charts. Linked brushing is one way to reintroduce some of that interconnectedness to your data displays. With linked brushing, a user selects a subset of values by dragging (or brushing) over the chart area. Then, any “linked” charts update automatically to reflect or highlight the same data, as shown below. A basic example of linked brushing. As a user brushes over the scatterplot of penguin bill dimensions (top), the histogram updates to highlight the body mass distribution for selected penguins. Linked brushing is a specific type of _cross-filtering_, a general term for filtering data displayed in one or more charts, based on the user’s selection in another. When there exist obvious or pre-defined categories, cross-filtering using simple selection (for example, by clicking on one wedge of a pie chart to filter to only that category) may be a better option — and it’s certainly more common in dashboards. But brushing can be particularly useful when you don’t want to limit a user to selecting a pre-defined category or region, giving them more control over which subset of the data they dig into. In this post, we highlight three examples that show how linked brushing can help users explore patterns across dimensions, dive deeper into local spatial trends, and focus on time periods of interest. For the right data, linked brushing can produce richer, more engaging visualizations that reveal relationships across metrics, without overcomplicating individual charts. ## **Explore across dimensions** Brushing can give viewers a fast, fluid way to investigate patterns across dimensions, making it a useful option for quick data explorations. For example, in the scatterplot matrix below, a user selects points (representing body size measurements for 344 penguins, including flipper length, body mass, and bill dimensions) by dragging the mouse over any area, in any chart. The records within the selected area are simultaneously highlighted across all charts One way this can be particularly useful is when you want to investigate outliers or anomalies you notice in the data. For example, if you notice a cluster of penguins with unexpectedly low body weights, you could brush over those points to reveal if the same individuals were noticeably smaller across other dimensions. A brushable scatterplot matrix can reveal patterns across different quantitative variables (and the relationships between them). Try it out yourself! Visit [this notebook](/content/@d3/brushable-scatterplot-matrix/index.html) to explore the chart above, and to see the code that implements brushing in the chart above using D3. ## **Dig into spatial patterns** With geospatial data, linked brushing can help a viewer to focus on events and patterns within and across regions of interest, and to see how those patterns change across another dimension (like time!). The example below, made with Mosaic vgplot, visualizes one million taxi pickup and dropoff locations, along with pickup times, recorded in New York City from January 1 to January 3, 2010. Brushing across values in the histogram, or over either map, will filter to only show the included records in both maps. Explore [this example](/content/framework/lib/mosaic#mosaic-vgplot/index.html) in our Framework documentation to see how spatial patterns in taxi rides change throughout the day. Brushable maps and histogram help a user to explore spatial and temporal patterns in taxi ride pickup and dropoff locations in New York City. Want to go beyond a simple rectangular brush? See example code for [lasso selection in D3](/content/@fil/lasso-selection/index.html), which lets you manually draw a custom shape to select values. As mentioned previously, brushing isn’t the only way to explore spatial patterns across different charts and metrics. Imagine if the maps in the example above included an additional layer with New York City neighborhood boundaries. Instead of brushing (which typically refers to a specific action where a user manually drags over an area of a chart to highlight the selected elements), a user might just click on any spatial polygon to focus on pickups and dropoffs within the borders of one or more neighborhoods. This action isn’t technically brushing, but _is_ a cross-filtering method that highlights the same data subset across charts, albeit one that limits user selections to pre-defined regions. And, it’s possible to build charts that combine the two, allowing a user to brush over discrete categories or regions, for example to [brush over ordinal categories](/content/@d3/ordinal-brushing/index.html) to select sequential groups. ## **Focus on time periods of interest** Brushing can also help a viewer to zoom in on interesting patterns in time series data, without needing to know or select specific start and end dates in advance. If you see an interesting event, change, or anomaly in time series data, brushing can quickly give a more detailed view of the surrounding period. The example below shows 30- and 15-year fixed-rate mortgage rates from [weekly Freddie Mac lender surveys](https://www.freddiemac.com/pmms/about-pmms) (ongoing since 1971). Brushing over the long-term time series chart produces a zoomed-in chart of mortgage rates within the selected window. This lets a viewer dig into short-term trends, while still showing how it fits into the bigger picture of the full dataset for useful context. A brushable line chart of mortgage rates ( [made with Observable Plot and D3](https://observablehq.observablehq.cloud/framework-example-mortgage-rates/)) allows a user to focus on a shorter time period of interest, while still seeing how it fits into the longer term pattern. As shown in the examples, brushing can be a useful option when you want to maintain simplicity of individual charts, while allowing a viewer to explore patterns or anomalies across metrics, dive into localized trends in spatial data, or zoom in on time periods of interest. For more inspiration, and to start building richer interactive charts, check out the example galleries for our open-source data visualization libraries: - [Observable Plot gallery](/content/plot/index.html) - [D3 gallery](https://d3js.org/) \]\]>https://observablehq.com/blog/truth-about-pie-charts https://observablehq.com/blog/truth-about-pie-chartsTue, 14 Jan 2025 00:00:00 GMTYou’ve heard the jokes about them, you’ve been told not to use them, maybe you’ve been made fun of for using one anyway: pie charts! They’re as popular with the masses as they’re hated by some. So what are pie charts good for, when _should_ you use one, and what do you tell the know-it-all who keeps insisting that they’re always the wrong choice? In this post, I will walk you through what pie charts are good for, how donut charts work, and finally take a look at what the research says (such as it is). Why should you trust me? I worked in visualization research for a while and have run studies on pie charts (I’m linking to a few of my studies at the end of this blog post). In fact, I might be the person who has run the most studies on them specifically. ## **It’s okay to dislike pie charts** Let’s get this out of the way first, though: I’m not here to sell you pie charts. If you don’t like them, don’t use them. What I am taking issue with, however, is that opinions and aesthetic judgments get passed around as facts. It might surprise you how little we actually know about pie charts, given how ubiquitous they are and how long they’ve been around. In fact, pie charts are among the oldest statistical charts that we recognize as charts. William Playfair is generally credited with their invention, in his _Statistical Breviary_ published in 1801. People seem to have disliked them for nearly as long. Willard Brinton, for example, stated in his 1914 book, [_Graphic Methods for Presenting Facts_](https://archive.org/stream/cu31924032626792), that “\[the\] circle with sectors is not as desirable an arrangement as the horizontal bar \[chart\].” And the list goes on. Pie charts aren’t always the right choice for the data or the question being asked. However, many of the claims people like to make about them are either wrong, or there’s no evidence for them. My goal here is to present what we know about pie charts (and what we don’t know!), so we can decide to like or dislike them on their actual merits. ## **Part-whole comparison** No chart is good for everything, it always depends on what your data looks like and what task you have in mind. Bar charts are great for comparing values to each other, such as returns from different investments, or the mileage you get with different cars. Direct comparison of values is a common task, so bar charts are often a good choice. But not always! Sometimes, the question is not comparison between values, but between the numbers and the sum of all the numbers. This is called part-to-whole comparison, and it’s what pie charts are good at. Examples include the contributions of different departments to the total sales of a corporation, or the breakdown of the U.S.’s 2015 population into ten-year age groups in the charts below. As you can see in the image, if you’re looking for direct comparison between values (e.g., which age group is the largest?), the bar chart is the way to go. But try to figure out what fraction 50-59-year-olds constitute, and you’ll have a much easier time doing that with the pie chart. This kind of comparison is often taken to show that bars are superior, but that’s only true if all you care about is direct comparison of values — then by all means, use a bar chart  or a table. There are many ways to compare values to each other directly. It’s also important to ensure that your numbers add up to 100%. If there are missing categories, or things can be counted in multiple places, the pie chart no longer works. A reason many people dislike pie charts is because they’re often used for data that doesn’t fit the chart’s underlying metaphor. There aren’t many other charts that allow a part-to-whole comparison in the same way the pie chart does. Two that come to mind are treemaps and stacked bars. Neither of which has been shown to be better than the pie chart in studies (see below), though — and the stacked bar chart is actually consistently worse. ## **Accuracy** One of the few pie-related things we have some data for, viewers' ability to accurately interpret and compare values shown on pie charts. A classic study in data visualization, Cleveland and McGill’s 1984 _Graphical Perception: Theory, Experimentation, and Application to the Development of Graphical Methods_, found that viewers were less accurate when comparing values in pie charts than when comparing bars located next to each other. However, when the bars were stacked or not placed directly next to each other, pie charts did as well or even better. Data visualization is rarely about precise comparison, however, and when exact numbers are called for, a table is usually the better choice anyway. For other tasks, such as getting a quick but rough idea of how much a department contributes to a company’s sales, pie charts are perfectly adequate. The difference in accuracy is usually less important than the mismatch between chart and task when using a bar chart for that same purpose. This is especially true when the values are different enough from each other that they are easy to tell apart (even if not with perfect accuracy). When the values are close enough that they can’t be distinguished from each other, this becomes a larger issue. This is usually not under the visualization designer’s control, unfortunately. ## **Pies and donuts** One of the common misconceptions about pie and donut charts is that donuts are worse than pies. This is based on the idea that we read pie charts by the angle in the center of the chart, and since donuts are missing the center, clearly they have to be worse! But as I describe in the research section below, no study I’m aware of has shown either of these two common assumptions to be true. Pie and donut charts aren’t read by angle (at least not alone), and donut charts don’t do worse in studies than pies. Donut charts aren’t very common when there are multiple categories, but they are often used to show how two numbers relate to each other, like in how close sales are to a goal or similar. That is also where it is often useful to see the exact number, which is then placed inside the donut hole. ## **What does actual research say?** First, it’s important to understand that most information about pie charts that is circulating online (and even in books) isn’t actually informed by research, but rather based on opinions and aesthetic judgments. That even includes the basic mechanism of how we read pie charts. When you draw a pie chart, you typically do so by angle: you divide the circle according to the values you want to show. If you’re dealing with percentages that add up to 100%, you can multiply each value by 3.6 (360º for 100%) and draw each circle segment accordingly. That leads to the assumption that we also read the chart like that, but there is no actual evidence for that. The problem is that pie charts use at least three different visual cues at the same time: angle, area, and arc length (as well as a few other lesser-known circle-related measurements). But which one do we use? While it seems that we should know how pie charts work, the truth is we don’t. The usual story about how pie charts are read by angle isn’t based on any studies. Walter Crosby Eells’ 1926 paper [_The Relative Merits of Circles and Bars for Representing Component Parts_](http://www.jstor.org/stable/2277140), is often cited as the source, but it doesn’t actually say that. Methodological issues aside (it uses self-reported cues, which are very unreliable), it finds that area is the most likely visual cue used for pie charts. In 2016, Drew Skau and I published [a paper that reported on a pair of studies](https://eagereyes.org/blog/2016/an-illustrated-tour-of-the-pie-chart-study-results) that tried to suss out how we actually read pie charts, which nobody in the 90 years between Eells and our study had bothered to do. What we found was that people were less accurate at comparing values when given just angles, and did much better with area or arc. We also found that donut charts did no worse than pie charts in terms of accuracy when reading values, which makes sense if angle isn’t used to read them (or just not important). I later [re-analyzed the results from an older study](https://eagereyes.org/blog/2016/a-reanalysis-of-a-study-about-square-pie-charts-from-2009#a-reanalysis-of-a-study-about-square-pie-charts-from-2009) whose purpose was to test something else, but where we also had used pie and donut charts as control conditions. It turned out that in that study, donuts also did exactly as well as pie charts (and stacked bars did worse). In a later study, I even [used 3D pie charts to try to separate area and arc length from each other](https://eagereyes.org/blog/2019/paper-evidence-for-area-as-the-primary-visual-cue-in-pie-charts). The results were again consistent with angle not being important, but there ultimately wasn’t a clear winner between arc and area. It seems possible that we use a combination of both visual cues, or that we switch between them depending on the size of the slice. Between the consistent donut chart results and the isolated cues, I am confident to say that we’re not reading pie charts purely by angle, however. ## **When to use pie and donut charts** Pie charts have their place in part-to-whole comparisons when the number of slices is small and the categories cover the entire space (i.e., they sum up to 100%). Donut charts work well to show goal achievement, resource utilization, etc., especially when they’re only comparing a single value to the total. They’re certainly no worse than pie charts. Pies are popular, they’re attractive, and for most purposes they’re accurate enough. Whether you use them or not is a matter of personal preference, not based on some supposed shortcoming. While there isn’t much research in this space, we should at least be aware of what’s out there and what it says. \]\]>https://observablehq.com/blog/kelsey-nanan https://observablehq.com/blog/kelsey-nananThu, 09 Jan 2025 00:00:00 GMTWhen talking with data visualization developer Kelsey Nanan, one thing quickly becomes clear: her path into this field wasn’t a traditional, carefully charted route. Instead, it began with a single elective class during her master’s program in Urban Science that sparked what she calls “an almost obsessive interest in web mapping” and eventually led her to work on complex, socially impactful projects. Hers is a story of how a single moment of discovery can reorient an entire career. “I took this one specific class during my master’s,” says Kelsey. “It was basically an intro to front-end development for people who were not technical at all. I just fell in love, and I became obsessed.” That obsession led her first into web mapping, which became the gateway into what she really loves: custom [data visualization](/content/learn/what-is-data-visualization/index.html). > “In data viz development, anything you can imagine, you can create,” she explains. “I really enjoy that. It gives you a lot of flexibility and creativity.” Kelsey Nanan It’s a mindset that has carried her through a variety of roles and projects. Early on, she “weaseled” her way into her first data visualization role where she used D3 to rebuild and customize charts at a startup, and that experience led her to later work at McKinsey’s Data Visualization Lab. More recently, she’s been bringing complex stories to life as a freelance data visualization developer. Throughout it all, Kelsey has continued learning, experimenting, and adapting — an ethos that shines through in the projects she highlights and the advice she shares. ## Evolving ideas with data One of Kelsey’s core philosophies is to let the data speak and shape the story, rather than forcing a predetermined narrative. This principle formed the foundation for a project she worked on called: [Streets to Stability](https://streetstostability.com/) that explores the ecosystem of homelessness services in San Francisco. What initially started as a request for a visualization that mapped service providers to different stages of homelessness rapidly revealed the daunting complexity of San Francisco’s support system. The system, designed to aid individuals experiencing homelessness, is convoluted: more than 230 providers receive public funding to offer healthcare, housing access, behavioral health, and social support, but these services frequently overlap. This leads to a tangled web of resources in which someone may have to navigate 15 to 30 different service providers to get the help they need. Kelsey describes it as the kind of work that began with one idea and evolved into something far more expansive and impactful than anyone had anticipated. “What it was initially conceptualized as was so different from what the end product ended up being,” she notes. The shift happened organically as the team dug into the data, revealing bigger stories about policies, barriers, and opportunities for change that were beyond the original scope of the project. “We first started sketching out the complex network of homelessness service providers and seeing what a mess it was,” Kelsey says. “Then we started asking questions: how can we fix this? Why is it like this? The research and data ended up shaping the direction of the whole project.” In the end, Streets to Stability became more than just a visualization. The final report offers policy recommendations and a fresh look at a complicated, human-centered issue. It proves that when you let the data guide the narrative, the resulting story can be richer and more impactful. Streets to Stability guidelines for improving the path from homelessness to housing. ## Behind the scenes: An array of tools and team efforts On the technical side, projects like Streets to Stability are far from one-tool affairs. Kelsey and her team combined a range of methods and platforms, from design tools like Figma and Flourish to front-end developer tools like [Observable Framework](/content/framework/index.html), [D3](https://d3js.org/), and even features in vanilla JavaScript. She also discovered new chart types along the way, such as [parallel sets](/content/@d3/parallel-sets/index.html), which she found through Observable’s [D3 gallery](/content/@d3/gallery/index.html). The result is a deeply collaborative process. Designers and developers iterated back and forth, sometimes finalizing charts directly in Figma, then integrating them into the code. That cross-disciplinary workflow helped the team stay nimble, especially when last-minute requests — a new splash page, additional introductory scrollytelling sections — arrived just days before launch. “You think it’s too good to be true when you have a whole week left to wrap up a project — and it usually is,” she says, laughing now about the eleventh-hour time crunch that ultimately led to crucial new storytelling components. Kelsey adapted the team’s original waterfall chart to this parallel sets chart, making the data more readable. ## Remembering the audience: Accessibility and clarity Kelsey’s approach to data visualization isn’t just about creating beautiful visualizations from complex information. She’s also keenly aware of the importance of audience, accessibility, and context, citing a project with the [Perkins School for the Blind](https://www.perkins.org/our-work/cvi/by-the-numbers/) as a formative experience. The goal was to present research on Cortical Visual Impairment (CVI) in a way that would be understandable and navigable by people with varying degrees of vision impairment, including those who rely entirely on screen readers. This challenge required rethinking how data-driven pages are structured. “If we were just creating this for another client, it might have been standard maps and hover tooltips,” she says. But for Perkins, the team had to include fully accessible tables of the underlying data, ensure keyboard navigation was fully supported, and provide narrative text descriptions of patterns on the maps. “Visually impaired people might need a description of what’s on the map,” says Kelsey, “and that also helps people who may not be very data savvy.” In other words, focusing on accessibility improves clarity for _everyone_. ## Advice on learning and growth For those intrigued by data visualization, Kelsey’s journey offers a few lessons. She’s the first to admit that her road to mastering D3 and other tools was nonlinear. “I kind of weaseled my way into my first data viz development job,” she says with a smile. “I had taken a course in D3, so I felt like I had a basic foundation, but I wasn’t sure it measured up to what the job really required.” But diving in headfirst taught her valuable skills like hover interactions, text wrapping, and dynamic scales, and gave her the confidence to try new things. As for learning new tools, Kelsey encourages people to find the method that works best for them. “I wish I had known a little more about the theory behind D3 earlier,” she admits. “But I’m the type of person who doesn’t stick with courses or tutorials for long. I learn best if I have something concrete I want to build.” Her recommendation? Understand your learning style. Project-based learning, trial and error, and following your curiosity all have a place. Equally important is understanding that what you think you want to do might change once you see your data. She warns against falling into the trap of deciding on a chart type or design before completing a full [data exploration](/content/learn/what-is-data-exploration/index.html). By staying flexible and letting the data guide you, you create more meaningful work. “Maybe your data isn’t a map. Maybe it’s a scatterplot or a beeswarm chart,” she says. Pen and paper sketches help her stay open-minded: “I’m not good at drawing, but I do it every time.” Some of Kelsey’s sketches. ## Infusing delight and staying inspired Kelsey measures success in data visualization by how much it sparks curiosity, encourages exploration, and leaves viewers genuinely excited to see more. She approaches her work with a simple litmus test: if she can’t stop playing with her own prototypes, there’s a good chance her audience will find them engaging, too. In her view, every interaction should spark curiosity rather than feel like a chore. “People have short attention spans, myself included,” Kelsey admits. “If I have to click around a lot, sometimes I don’t want to be on that page anymore.” By embracing smooth hover interactions, intuitive layouts, and non-traditional chart types, like parallel sets instead of waterfall charts 😉, she keeps the door open to discovery and learning. What emerges is a cycle of motivation and growth. The joy Kelsey infuses into her own work keeps audiences engaged, and the inspiration she draws from others encourages her to test new mediums, forms, and technologies. In this iterative process, data visualization evolves from a technical discipline into a playful, human-centered art form — one that nudges viewers to linger, learn, and ultimately see the world in a new way. As a perfect example of this, she mapped the lyrical themes and “singability” of each track on Taylor Swift’s _The Tortured Poets Department_ onto a [vinyl-inspired radial chart](https://nightingaledvs.com/interactives-taylor-swift-data-visualization/). Kelsey’s visual analysis of Taylor Swift’s The Tortured Poets Department. By anchoring on data, delight, and the human connections behind every project, Kelsey demonstrates the profound impact that thoughtful, well-crafted visualizations can have. It’s a reminder that data isn’t just numbers — it’s stories, people, and possibilities waiting to unfold. _For a closer look at Kelsey’s collaborative approach,_ [_watch her talk from Observable’s Insight 2024 conference_](https://www.youtube.com/watch?v=gRBVR-PI6pw) _. In it, she shows how building bridges between designers, developers, and stakeholders leads to more engaging, effective visualizations that resonate with audiences and drive meaningful outcomes._ \]\]>https://observablehq.com/blog/observable-wrapped-2024 https://observablehq.com/blog/observable-wrapped-2024Tue, 17 Dec 2024 00:00:00 GMTFrom new open-source software, to inspiring community creations, and an exciting conference, it’s been a big year for Observable. As we look back on 2024, here are 24 insights from the year. ## **Insight 2024** Our first insight is quite literal: the [_Observable Insight_](/content/insight/2024/index.html) conference we hosted earlier in December. Over 500 data viz developers, data scientists, analysts, and product managers joined us live for inspiring talks, discussions, demos, and creator spotlights showing what’s possible (and what’s coming!) in [data visualization](/content/learn/what-is-data-visualization/index.html). If you missed _Insight_ or want to revisit any of the insightful sessions, [all recordings are available in our YouTube playlist](https://www.youtube.com/playlist?list=PLOHIJAFwtkEeMHFwEoMX-6a4zt6gd2-d7)! ## **Twelve community insights** Observable community members created many amazingly insightful data visualizations and data apps this year. Picking just a handful to highlight was almost impossible, but here’s a dozen we particularly liked. Since releasing [Observable Framework](/content/platform/framework/index.html) in February, we’ve been wowed by the dashboards and data apps built with it by the community. Here are a few that caught our eye: - [Climate Central’s analysis providing insights into urban heat hot spots](https://climatecentral.observablehq.cloud/urban-heat-hot-spots/) (see also [the customer story](/content/customer-stories/climate-central/index.html) for a look into how their team uses Observable) - [Streets to Stability, a very insightful piece on homelessness cycles in San Francisco](https://streetstostability.com/today) (all puns aside – see also [our recent blog post interviewing Jason Forrest, one of the project’s leads](/content/blog/jason-forrest/index.html)) - [Private jet emissions (and why they are bad)](https://csdiehl.github.io/private-jet-emissions/) provides an interesting view into the flight patterns of the super-rich - [Mosaic + Framework](https://idl.uw.edu/mosaic-framework-example/) makes it possible to visualize very large data sets to gain insights into them - [A map inventorying Boston’s buildings](https://m-clare.observablehq.cloud/boston-building-retrofits/) to investigate their potential for reducing carbon emissions through retrofitting - A [real-time view of the BlueSky firehose](https://mclare.blog/posts/visualizing-the-blue-sky-firehose/), giving an overview of the most popular hashtags on the platform Our users also created many remarkable notebooks providing insights into a wide variety of topics. Here are a few that we love: - [A very insightful presidential election cartogram transition animation](/content/d/3e39dc77f8af714a/index.html), which unlike most preserves geographical relationships - [The Tortured Data Viz Department](/content/d/fc3dbf7fae601f76/index.html) – yes, it’s about Taylor Swift, of course - [Great circle vs. rhumb line comparison](/content/@neocartocnrs/great-circle-vs-rhumb-lines/index.html), showing graphically why geometry on a curved Earth is different from a map - [Jo Wood’s 30-day map challenge series](/content/@jwolondon/tdmc2024_day15?collection=@jwolondon/30-day-map-challenge-2024/index.html), which includes many beautiful, and no less insightful, maps - [Movie monster beeswarm plots](/content/d/956c35b7a3867887/index.html) providing a view into the popularity of different kinds of monster movies (and movie monsters) - No year-end “wrapped” post is complete without some insight into Spotify data. [The relentless march to musical oblivion](/content/@bartok32/the-relentless-march-to-musical-oblivion/index.html) shows how songs become popular and then fade from people’s attention over time. ## **Ten insights into making better data apps** Helping people gain new insights into their data is a core part of our mission. Going beyond our product releases, we also created [blog posts](/content/blog/index.html), held [webinars](https://www.youtube.com/@Observablehq/streams), gave external talks, and published a number of [videos](https://www.youtube.com/@Observablehq/). In particular, we wanted to help people create better data visualizations, by discussing [why you should never crop the y-axis, and when you still might](/content/blog/never-okay-crop-y-axis-except-when-it-is/index.html), [showing you how to do it in Observable Plot (and what to do instead)](/content/blog/why-never-crop-y-axis-part-2-how-to-and-what-to-do-instead/index.html), [talking about underutilized chart types](/content/blog/five-underused-charts-for-richer-dashboards/index.html), [how to add annotations to your charts](/content/blog/five-techniques-to-improve-chart-annotations/index.html), and [how to use animation in your visualizations](/content/blog/effective-animation/index.html). Beyond data visualization, we also dove deeper into how to build better data apps, including [why data needs to be updated with care](/content/blog/align-dashboards-with-scales-of-business-decisions/index.html), [how code opens up many possibilities for dashboards and data apps](/content/blog/three-ways-build-better-dashboards-data-apps-with-code/index.html), and [how a static-site generator works](/content/blog/what-is-static-site-generator/index.html). Rounding out this section, we also gave talks and workshops at external events, where we shared how Observable's open-source tools help folks create better data visualizations, faster. For example, check out [Robert’s talk at DuckCon, _A duck for your dashboard: performant data apps in the browser with DuckDB_](https://www.youtube.com/watch?v=blYQhiOMhwA), and [Allison’s session at OSA CON on how developers can build faster and overcome the last mile problem](https://www.youtube.com/watch?v=KynLyg5qMWM). ## **An insight into our own open-source projects** Finally, we wanted to gain some insight into our own work. Between all our projects and products, we closed over 1,600 tickets this year, and made almost 6,000 commits across all our repos. Our open-source projects, including [Observable Framework](/content/framework/index.html), [D3](https://d3js.org/), and [Observable Plot](/content/plot/index.html), are being used by millions of people across the globe. But we wanted to know if people mostly use them for work, or also for their personal projects. Our [open-source analytics project](https://observablehq.observablehq.cloud/oss-analytics/) provides download numbers for all our projects, which allowed us to quickly create the visualization below. It shows download numbers for each project over the last six months on circular heatmaps. These align days of the week (each spiral layer is one week), so it’s easy to see that the numbers are a fair bit lower on the weekends (lighter wedges on Saturday and Sunday) than during the week (generally darker wedges Monday thru Friday). ## **On to 2025** We hope you had an equally insightful 2024 and wish you a healthy and successful 2025! There will undoubtedly be many more incredible visualizations and data apps built by the Observable community, and we’re looking forward to seeing them. We also have some great new things planned for the next year, so stay tuned! \]\]>https://observablehq.com/blog/jason-forrest https://observablehq.com/blog/jason-forrestThu, 12 Dec 2024 00:00:00 GMT_Note: This is our third post in a series highlighting the work, experiences, and perspectives of leaders in data visualization._ It’s late September when we first reach out to Jason Forrest about an interview, and he’s neck deep in putting the final touches on a new exhibit, [_Hello from the Data Vandals_](https://datavandals.com/hello-from-the-data-vandals), which opens days later at Cooper Union in New York City. The show — a joint effort by Jason and artist [Jen Ray](https://cargocollective.com/JenRayArt), the duo behind [The Data Vandals](https://datavandals.com/) — explores data on housing, environmental issues, food, and demographics in the East Village neighborhood through art, data visualization, film, workshops, and panel discussions. Jason Forrest mounts pieces for the [Hello From the Data Vandals](https://datavandals.com/hello-from-the-data-vandals) show at Cooper Union, New York City. He sends along a short video of the exhibit space, filmed on his phone while walking past bold pop art, hand-painted statistics, and data viz projected onto a back wall. The final seconds of the clip zoom in on a large yellow poster with “WHO THE HELL ARE THE DATA VANDALS?” painted in all caps across the top. A brief description follows: “Our society looks at the world through data, but data doesn’t have to live in a dusty database! We create a new kind of art to begin conversations that affect us all.” When we meet with Jason the week after the show, we quickly learn that _beginning conversations_ isn’t just a Data Vandals tagline — it’s core to Jason’s work and worldview, and not only as a data visualization designer. Whether through art, music, publishing, photography, or data viz, Jason’s on a mission to give people something to talk about. “That's where the art is,” he says. “It’s in the dialogue and the discourse.” ## **It’s the dialogue, stupid** These days, Jason Forrest is a big name in data visualization. He led data viz efforts at McKinsey & Company — one of the largest consulting firms in the world — as a designer, analyst, and ultimately Director of the McKinsey Data Visualization Lab. In 2019 he co-founded [Nightingale](https://nightingaledvs.com/), the Journal of the Data Visualization Society, for which he still serves as editor-in-chief. He established the biannual print spinoff, [Nightingale Magazine](https://shop.datavisualizationsociety.org/?gad_source=1&gclid=CjwKCAiAl4a6BhBqEiwAqvrquvfaDFs9WaWrOILLMVqYhZu5LqoH9MIOM6JSjjPcjLa-iTcB-nkGLxoCdDEQAvD_BwE), in 2022. And, earlier this year, he founded the [Jason Forrest Agency](https://jasonforrestagency.com/) to build tools that help clients “explore data for better, faster decisions.” But Jason has been doing work that inspires conversations since long before he got into data visualization, even if he didn’t realize it at the time. “I've always been interested in bringing people together to enjoy some shared sense of community about something we're interested in,” he reflects when we ask what ties his creative and professional experiences together. “That could be data viz, that could be skateboarding, that could be music, that could be history. But it’s only been in the last five years that I realized the throughline was the dialogue.” Jason’s unconventional path into data viz includes formal training in photography, a five year span as an artist and art critic, and a nearly two-decade-long stretch spent producing and performing electronic music that is described on the [Jason Forrest Wikipedia page](https://en.wikipedia.org/wiki/Jason_Forrest) as “noisy experimental electronica and [breakcore](https://en.wikipedia.org/wiki/Breakcore)”. It was his music career — the tail end of which overlaps with his foray into UX design — that put Jason on a collision course with [data visualization](/content/learn/what-is-data-visualization/index.html). > “Through my years in music, I realized that all of your creations and your creativity are a dialogue with the works that you've been inspired by, and also a dialogue with the people who consume and discuss your work. That's why I really fell in love with data visualization. It has all of the elements of creativity, community, and technical complexity that I found in music and art and design. They were all there in data viz.” Jason Forrest Jason DJs a show in Berlin, 2003. Photo: [jasonforrestftw.com](http://jasonforrestftw.com/). If dialogue was a side effect of his past creative work, it’s front of mind for Jason today. In fact, he’s made dialogue _the point_ of his current professional and creative work. He shares another story, this one from a 2022 Data Vandals project on gun violence. “It was in Central Park. We had put up all these horrible statistics on gun violence, and it was right after the Uvalde, Texas mass shooting. We went out there thinking that we would get people yelling at us because, you know, America — it’s a crazy place. Instead, everybody just wanted to talk. We learned once again that the conversation _was_ the art.” It’s a lesson he has carried forward in his dataviz work, through Data Vandals and beyond. “Ever since that point, we've tried to structure everything we do around data as a way to welcome people in, and see themselves in the data. It’s fascinating to see how many people, when you open the door and say ‘Come on into the conversation,’ will join you.” The Data Vandals’ [American Monument](https://datavandals.com/american-monument) memorial explores gun violence in the United States (May 2022). Photo: The Data Vandals. ## **Empathy, excellence, and tuning the hi‑hat** Designing empathetic visuals and user experiences that open a door into the data is a recurring theme in our chat. So, we were surprised to learn that empathy wasn’t always a priority in Jason’s data viz approach. In fact, early on he viewed data work as a way to remove human biases and emotions from decision-making — and he was determined to make viewers stick to the cold, hard, data-driven facts. “I wanted to get involved with data viz because misinformation was on the rise, and I thought the only way to combat misinformation was with the idea of statistical truth,” he recalls, before adding (with a knowing smile): “You can't argue with the numbers, right?” He quickly accepted the futility of trying to human-proof data visualizations. > “Of course, we all know you can argue with the numbers. So, trying to fight misinformation meant that I actually had to learn more about how people connect with data, instead of trying to separate them from it.” Jason Forrest For Jason, connecting people with content requires first understanding the audience. He takes us back to his DJing days in Berlin, circa 2003: “When I used to put on rave parties, you had to put posters on the _right_ street corners at the _right_ time to get the _right_ audience to see it. If you did it an hour before, then it would be seen by business people — not the people that would come to your concert. If you did it two hours later, other people would put their posters over yours. So you really had to get the timing right, and make it very customized to the audience.” The takeaway: whether you’re promoting a techno show or coding up a chart, you’ll miss opportunities to connect with viewers if you don’t design with them in mind from square one. And when it comes to designing engaging and impactful visuals, Jason doesn’t think any detail is too small to warrant attention. “Back when I was a musician, I used to call that ‘tuning the hi-hat’,” he shares, describing the practice of tweaking the finest details even beyond what a casual viewer might notice. (For context, the [hi-hat](https://en.wikipedia.org/wiki/Hi-hat) is the tall double cymbal common in many drum sets.) “You're literally trying to find a pitch that matches the tonality of the hi-hat, which is largely atonal. The point is, you're doing something that no one's really going to see. You’re fussing around with the language of a footnote, or the formatting of a table that’s hidden in a deep link.” Where others might see this as attention wasted on barely-there details, Jason sees it as an essential part of data viz development. ”The catch is, that’s where the magic is. That’s how we reach toward excellence — by tuning the hi-hat.” And tune, they do. The attention to detail and user experience is obvious throughout Jason’s data viz portfolio, including in [Streets to Stability](https://streetstostability.com/), a recent report built with [Observable Framework](/content/framework/index.html) that explores cycles of homelessness in San Francisco. [Streets to Stability](https://streetstostability.com/) combines illustration and data visualization to explore cycles of homelessness and support options in San Francisco. There’s no one thing that makes Streets to Stability impressive. Stunning charts are presented alongside hand-drawn illustrations that humanize complex data. Interactive charts pair with bite-sized anecdotes that ground the data in possible real-world outcomes for unhoused individuals in the city. And scrollytelling draws in the viewer, then shepherds them through the piece. An overarching strategy that makes it so effective, Jason shares, is the careful integration of creative and technical elements. > “The illustrations help to empathize with and contextualize your work. The data helps to provide that statistical truth and authenticity that underlies the story and the conversation. It's really the pairing of the two contexts that fulfills the story.” Jason Forrest ## **Data visualization as a team sport** The pairing of creative and technical content is so powerful that Jason has built it into his data viz development process. “What I've done with my teams over the last few years is to pair someone who has more of a design specialization with someone that has more of a specialization in the tech. That could be a UX designer and a data scientist, or a data viz designer and a coder, or a subject matter expert and someone who's a data engineer. That way there's someone who can penetrate into the technology and the data, and there’s someone who can penetrate into the story and the concept.” And even if someone _can_ do it all, Jason prefers collaborative work to solo efforts. > “Yes, there are unicorns out there that can do both. But to be honest with you, everything works better as a team sport. In the act of talking to people, you learn new things, and then you can create something that's going to be inherently more communicative — and that's the only thing that matters.” Jason Forrest He shares his screen to show us a visual overview of the methodology he’s developed to get data visualization projects from a highly uncertain exploratory phase, to a more certain(ish) final product. Unsurprisingly, it’s titled _Our methodology is based on collaboration_. Jason’s four part methodology for data visualization projects, from ideation to final product. Jason walks through the schematic, quickly touching on the exploratory, design, and building phases (“at some point you just have to focus on doing the damn thing”). Then, he spends a bit more time mulling on what he refers to as the “last phase of data visualization” — handing off the final product. It’s a step, he says, that’s critical to a project’s success, but too often overlooked. “One thing that's super important, and frankly almost a total blind spot in software development, is the documentation and training phase. There's a whole phase where you’re just handing off what you’ve made to the people who are going to maintain or market the piece. Training is equally important, and very few teams take the time to ensure their audience has been trained to use their work.” Jason talks about the audience (he takes care to use “audience” here, instead of “end users,” adding “we don’t really _use_ data viz, we _read_ data viz”) as equal collaborators in the data viz pipeline, almost as if they’re an extension of his ownteam. Explaining why he’s so invested in this final handoff, he brings us back to throughline spanning all of his creative and professional work: fostering dialogue. “If no one can use your work, it means no one will be aware of it. And if they’re not aware of it, then it's not generating a conversation. And then…what good is it?” _We’re sincerely grateful to Jason Forrest for sharing his time, perspective, experiences, and expertise with us. To learn more about Jason, visit_ [_jasonforrestftw.com_](http://jasonforrestftw.com/) _._ _Learn how Observable can help your team build and share custom, interactive data visualizations and data apps:_ [_observablehq.com_](/content/site-root.html) _._ \]\]>https://observablehq.com/blog/effective-animation https://observablehq.com/blog/effective-animationThu, 05 Dec 2024 00:00:00 GMTAnimation can enhance [data visualization](/content/learn/what-is-data-visualization/index.html) in many ways, by helping viewers make connections, by showing them different perspectives, and by just drawing them in to take a closer look. Animation can be subtle and act in a supporting role, or it can be the main feature of a visualization. It can also be polarizing, and with good reason. Used carelessly, animation can add unnecessary complexity and distraction, and turn into a gimmick. That is why it is often avoided in _serious_ visualization – but by doing so, visualization developers and designers lose out on its potential benefits. In this post, we share five ways that animation can improve data visualizations. For each, we describe how it can be useful for viewer understanding and experience, highlight published examples showing the approach used effectively, and point to ways to achieve similar effects with [D3](https://d3js.org/) or [Observable Plot](/content/plot/index.html). ## **Tweening for a smooth viewer experience** It is sometimes interesting to show different kinds of representations of the same or related data. This can be hard to communicate to the viewer, however. It can also be an opportunity to explain a more complex or unfamiliar chart type by building on a more familiar one. One way to do all these things is to morph, or _tween_, between different views. Tweening is a term from animated movies, and comes from junior animators drawing the frames in be _tween_ the ones drawn by senior artists. In data visualization, it usually means creating a smooth transition between different settings, shapes, or entire visualization types. Take this example transitioning between [vote counts for the winning party in each electoral district in the German state of Brandenburg](https://interaktiv.tagesspiegel.de/lab/landtagswahl-brandenburg-rote-staedte-blaue-land/). It shows votes gained by the AfD party (in blue), number of votes in each district (as circles), votes gained by the SPD party (in red), and votes lost by the Green party (in green). While the base map stays the same for context, the way data is shown changes from a choropleth map showing data as color, to spiky shapes using length/height, to circles using area. Another great example of tweening is this data story called, [Is the love song dying? by _The Pudding_](https://pudding.cool/2024/11/love-songs/). It starts out with a collection of dots for specific songs, then filters to a smaller selection of serenades that are put into a beeswarm plot on a time axis. From there, it creates a stacked area chart by adding all the other Billboard Top 10 hits from that same time period. After a few more steps that add songs about heartache, pursuit, things being complicated, and more, we end up with a bump chart! Tweening can also translate between types of maps. After elections, especially in the U.S., people often create maps that represent vote counts rather than area. These can be difficult to navigate though, because the distortion naturally shifts parts of the map around. A transition between the two can help preserve context, even when the map has been carefully designed like in [this example using data from the 2024 presidential election](https://bsky.app/profile/karimdouieb.bsky.social/post/3lacrji4c4p2k). These are some of the more elaborate examples of tweening, but it can also be useful for much simpler transitions, similar to the group comparisons in the previous sections. If you want to use it yourself in a D3 application, there’s the [shape tweening function](/content/@d3/shape-tweening/index.html) for this purpose. ## **Comparisons between groups or across metrics** There’s usually more than one way to sort data, and in some cases it’s useful to let your viewer see how the ordering updates over time or across different metrics. Imagine sorting a series of lines or bars by different criteria. Animated sorting can help viewers more easily track a group or series of interest as they toggle between metrics, providing visual continuity that makes a series or group easier to follow. The visualization below from the New York Times highlights [Tom Brady’s performance](https://www.nytimes.com/interactive/2022/02/02/upshot/tom-brady-career-stats.html) with smooth animation (in combination with thoughtful use of labels, stroke width, and color) as a user toggles between playoff wins by player age or year. Racing bar charts are another way that animation can add visual interest and help users track changes in relative position over time. They consist of a sorted bar chart to which data is added over time. The changes are animated, which makes it possible to see how they change relative to each other, and in particular when they change position in the sort order. Below is an example, created by Catherine Ramsdell and recently published in [Nightingale](https://nightingaledvs.com/racing-bar-charts/). It shows gross earnings for Broadway shows from 2001 - 2024, played back over time. Notice, for example, how easy (and frankly, just kind of mesmerizing) it is to track Wicked’s gross earnings once it appears in 2004. Of course, racing bar charts have earned some well-deserved criticism (even [banishment](https://builtin.com/data-science/bar-chart-races)) for their overuse and general lack of novel insights. But their longevity and popularity on social media, Reddit threads, and blog posts make a strong case for the value of animated data visualization _simply because it can be fun to look at._ Want to build animated charts like the ones above using D3? See and fork our reusable code for [racing](/content/@d3/bar-chart-race/index.html) and [sortable bar charts](/content/@d3/bar-chart-transitions/2/index.html) to get started. ## **Following spatial shifts** Maps naturally lend themselves to showing spatial data changing over time, even when they’re not being distorted. While this effectively shows motion as motion, it can work on a few different levels. Perhaps the most straightforward way to show data on a map is to show individual locations. [This animated map of the growth of Walmart over almost 50 years](/content/@d3/walmarts-growth/index.html) places Walmart locations on a map of the contiguous U.S. over the course of a few seconds. This reveals where the chain originated and how it spread, first over a small area, then across the entire country. [This scrollytelling piece about bird migration](https://www.reuters.com/graphics/HEALTH-BIRDFLU/MIGRATION/movaqmblrva/) uses motion to animate the counts of birds in different regions over the course of months and seasons. This lets the viewer get a sense of the migration patterns of mallards and how they change over time. This example doesn’t show individual birds, but works on a higher level of abstraction and aggregation. Another example of spatial data over time is [this map of forecasted precipitation during Hurricane Milton](https://www.ft.com/content/df240556-a957-4103-9e65-c51d8abf49ca). It depicts the amount of predicted rainfall across two days as a sped-up animation on a map. The result is straightforward, easy to read, and makes areas with more expected rainfall easy to identify. Finally, navigating between locations can itself be done using animation. The [D3 world tour](/content/@d3/world-tour/index.html) jumps between locations on a 3D globe and animates each hop. This makes it easy to always stay oriented about where each country is located and how far apart they are. ## **Motion as a visual cue** Sometimes, motion itself can be a useful visual cue that helps viewers get a better sense of the uncertainty in data. The most (in)famous use is in election forecasts and in particular the New York Times’ election needle. It shows the likely outcome of an election along a scale and encodes the uncertainty of the forecast in how smooth or jittery its motion is. [iframe](https://www.youtube.com/embed/ePpSYA9deKA?si=11wpXqzLmh887RgD&start=252) Another example of the same visual cue and use for elections, though used slightly differently, is [this visualization of the simulated outcomes of an election in Germany](https://www.zeit.de/politik/deutschland/2024-08/landtagswahl-ostdeutschland-afd-bsw-umfragen). The points for each party move between the predicted outcomes, which creates more jitter where the range is larger and less where it is smaller. Motion isn’t only used to show uncertainty, however, it also helps viewers trace the paths of data points through a complex shape. In an unusual diagram, [this piece in the New York Times](https://www.nytimes.com/interactive/2018/03/19/upshot/race-class-white-and-black-men.html) shows the transition of Black and white boys from their parents’ households to their own as they grew up and whether they moved up or down in income level. The way they show this is an unusual animated [Sankey diagram](/content/@d3/sankey/2/index.html), which draws a dot to represent 100 boys in the study and moves them from their parents’ income group to their own as adults. Race is encoded in the dots’ color, which allows viewers to directly see how the different points end up following different trajectories. ## **Revealing trends over time** Perhaps the most famous use of an animated chart is [Hans Rosling’s TED talk from 2006](https://youtu.be/hVimVzgtD6w?si=g8NYdaRw0g9J-BPU). He used a custom program called gapminder that created a chart that had not been in wide use until then: an animated scatterplot. It showed GDP per capita on the horizontal axis, and the percentage of child survival on the vertical axis (as a measure of health). Each country was represented by a bubble, sized by its population. What drew people’s interest was his use of animation to show how the world had evolved from 1960 to 2003. [iframe](https://www.youtube.com/embed/hVimVzgtD6w?si=SyvBpZ0q9FQr62-3&start=210) Rosling’s talk became iconic and made a large impression on people back in the early 2000s because of the novelty of the animated scatterplot. Today, it is easy to create animated scatterplots in most tools – including in D3 (see for example [Mike Bostock’s _The Wealth & Health of Nations_](/content/@mbostock/the-wealth-health-of-nations/index.html)). Some of the other examples we showcase above also fall into this category, such as the Hurricane Milton precipitation map. ## **Learn more** Motion is a powerful tool for data visualization. It can help people understand, prevent them from getting lost, and even encode data itself. It is best used sparingly, but when it is used with care, it can be incredibly powerful. Want to add animation to your toolkit? Here are some places to start: - [D3 gallery — animation examples](/content/@d3/gallery?utm_source=d3js-org&utm_medium=hero&utm_campaign=try-observable#animation/index.html) - [D3 documentation](https://d3js.org/) \]\]>https://observablehq.com/blog/announcing-embedded-analytics https://observablehq.com/blog/announcing-embedded-analyticsTue, 19 Nov 2024 00:00:00 GMT**Update (April 2025):** Observable Cloud is no longer available to new users, and embedded analytics as described here has evolved. We’re reimagining embedding as part of our new product, Observable Canvas. Displays of data have the power to deepen understanding, foster consensus, and shed light on hidden areas of opportunity. The challenge is getting that data in front of users in a way that is both accessible and digestible. Our newest capability allows you to bring rich, interactive data visualizations to your users wherever they are — including in an existing host application — without rewriting your entire application or building a stand-alone data app. With Observable’s new [embedded analytics](/content/documentation/data-apps/embeds/index.html) solution, it’s now easier than ever to integrate visualizations and dashboards built with Observable Framework directly into your applications without compromising on richness of data displayed, end-user performance, or brand continuity. ## The Observable advantage — embedded in your host application When we [launched Observable Cloud](/content/blog/announcing-observable-cloud/index.html), we set out to solve two key challenges in data visualization: first, the architectural challenge of getting rich data visualizations to load instantly without costly infrastructure; and second, the organizational challenge of enabling data teams to work more independently, free from complex pipelines and with fewer dependencies on external support. Our support for embedded analytics builds on these solutions, bringing the advantages of Observable’s baked data architecture into your application. ### Baked data architecture Historically, data teams have had to choose between expressive visualizations and end-user performance, especially when rendering large amounts of data in the browser. Observable’s baked data architecture is purpose-built for the web, delivering highly performant, expressive data apps, dashboards, and visualizations. Precomputed snapshots ensure that displays of data load instantly, while flattened transitive imports eliminate request chains, allowing embeds to load in parallel. This means your visualizations don’t just load faster — they perform better, scaling up to more complex data needs (i.e. larger data sets that display significantly more points of data) without the typical performance trade-offs. It’s an approach that also makes it possible to leverage the power and flexibility of [Observable Framework](/content/framework/index.html) within existing host applications without the heavy infrastructure costs or implementation delays for rewriting your existing codebase. ## Turn your analytics into a competitive advantage Request a demoSign up now ## A refreshingly simple integration Unlike other embedded solutions that often rely on SDKs, API keys, or iframes, Observable’s embedded analytics are simple vanilla JavaScript modules. Integrating them into your host application is straightforward — just import an [exported module](/content/framework/embeds#exported-modules/index.html) from an Observable Cloud-hosted Framework data app, and you’re ready to go. ``` ``` Observable Framework supports multiple integration options, including exported files (for hotlinking images, data, and other assets) and iframe embeds. But exported modules are the best way to ship rich, interactive embeds that integrate natively into your application. ### JavaScript modules for embedded analytics When building support for embedded analytics, we took the opportunity to test our solution for our own [project analytics](/content/blog/how-to-track-dashboards-enhanced-analytics/index.html) feature. Through this, we experienced firsthand the importance of ensuring that embedded analytics look and feel native to the host application. To make embedded analytics blend naturally into your application, exported modules allow visualizations to: - **Inherit styling:** Your visualizations can adopt the colors, fonts, and other styles within your application for consistent UX and easy code maintenance. - **Adjust responsively:** Embeds dynamically adjust to fit any screen size, adjusting to the viewport on resize and across a wide range of devices. - **Enable interactivity:** Users can interact directly with the visualizations, triggering actions through event listeners for enhanced functionality and seamless feel. And, because you and your team deserve analytics for your embedded analytics (yes, we’re talking metalytics 👍), export analytics let you see how often each module is viewed across your application, helping you monitor engagement and understand which modules are accessed most frequently. ### Personalized and secure data experiences at scale When developing user-facing experiences, it’s paramount that the correct data is shown only to the correct, authorized users. Observable’s embedded analytics support fine-grained control over data visibility, backed by standard JWT authorization. With parameterized routes, you can control access down to individual views, ensuring users see only the insights they’re meant to, consistent with your app’s permission structure. - **Personalization:** Use parameterized routes to tailor charts and visuals to each user, providing unique insights at scale. - **Multi-domain support:** In-app CORS configuration for Enterprise workspaces simplifies cross-domain embedding, allowing you to define access controls across domains for secure and personalized data experiences. - **Fine-grained access control:** Use secure access tokens and JWT-based authorization to control exactly what data is visible to each user, with permission logic controlled by your host application. ## Empower data teams and increase velocity Building new data experiences, or iterating on existing visualizations, can be an expensive time-consuming endeavor. To help accelerate your time-to-market and increase developer velocity, our approach allows data teams to operate more independently, iterating on insights without needing to touch the main application’s codebase. There’s no waiting on CI cycles or wrestling with a monolithic repository. This means updates to analytics are faster, more fluid, and less dependent on the core application’s development flow. ## A new era for embedded analytics Observable's embedded analytics offer developers and data teams a solution uniquely tailored for today’s data-rich applications. By combining instant-loading visualizations, independent data team management, and deep integration within your application, Observable brings embedded analytics up to the standards of modern, high-performance web development with no compromises. For a deeper dive into Observable’s approach and to explore embedded analytics in action, [visit our overview](/content/solutions/embedded-analytics/index.html). \]\]>https://observablehq.com/blog/ben-welsh https://observablehq.com/blog/ben-welshTue, 12 Nov 2024 00:00:00 GMT_Note: This is the second piece in our series highlighting the work, experiences, and perspectives of leaders in data visualization and analytics._ When we hop on a video call with Ben Welsh in late October, I’m immediately impressed by Ben’s background, blurting out even before we share introductions: “Is that your real office?” He chuckles and explains the scene — a busy array of overlapping highrises and ongoing construction just across 7th Avenue from his office, where he works as Data Editor at Reuters in New York City. “You’ll know that one. That’s One Times Square, where they drop the ball on New Year’s Eve.” He points to a section under heavy development, where work on a new [public observation deck and museum](https://www.archpaper.com/2024/04/renovation-one-times-square-curtain-wall-observation-deck-immersive-experiences/) is underway. “People can’t go up there now, but they’ll be able to soon. It’ll be open to visitors.” Ben Welsh in his office, with a view of construction underway at One Times Square. It’s a fitting backdrop for the discussion that follows, which focuses largely on building things that help people better observe and understand the world around them. Ben spends his days building things, too. But instead of new observation decks and experiences for tourists, he designs and engineers back-end data pipelines that support data journalism. “My role here as a Data Editor is to try to get more and more data out to our readers via our major outputs. So I am constantly building data pipelines, editing, and refining them down to stuff that I think might be interesting to people, and trying to get it out into the world.” He pauses for a moment, reflecting on his response. “That was really long-winded. I use data and computers to find and tell stories. How about that?” ## **Building automated data pipelines to make time for more important work** Ben has been using data and computers to help tell important stories for decades. Prior to joining Reuters he spent 15 years working at the Los Angeles Times, leading and growing their award-winning Data and Graphics Department. He’s contributed to high visibility pages and stories, including the LA Times’ first live election results, an acclaimed wildfire tracker app, and “the most complete resource on the spread of [COVID-19](https://www.latimes.com/projects/california-coronavirus-cases-tracking-outbreak/) in California.” Less publicly visible, however, has been Ben’s work to build data pipelines and tooling that improve how those stories are created and shared. Lucky for us, he’s thrilled to bring his back-end work to the forefront in our conversation. We’re meeting one month after Hurricane Helene devastated parts of the southeastern U.S. He uses the storm coverage as a jumping off point to describe his day-to-day work, and the benefits of automated data pipelines. “Every time one of these events happens, it’s big news. And, every time, there are things that we repeat as part of the news coverage, like making hurricane maps so that people can see the forecast and storm path, understand the severity of the storm at different times, and know whether they need to move.” That manual time and effort to process real-time (or nearly-real-time) data is where Ben sees opportunity. “In the past, newsrooms may have had a specialist whose job it was to download the data files and stick them in some sort of editing tool and manually convert them into a map. But in today’s world, we can automate that. A recent task we worked on was to write a software pipeline that can discover, download, process, and refine storm data, then generate those hurricane maps within seconds of new data being released. Then, after some human review, they are sent out around the world via our different publishing streams. The same approach can work for really any newsworthy data source that has a flowing stream of information.” One output of this effort is their new [hurricane map template](https://twitter.com/palewire/status/1853845161794486686) that compares forecasted paths from NOAA’s National Hurricane Center with other models. It’s powered, in part, by a [library to access data from the Automated Tropical Cyclone Forecasting System](https://palewi.re/docs/atcf-data-parser/), which Ben developed while “learning how to parse weird government data files.” Ben recently shared a new Reuters map template that compares hurricane forecasts from the National Hurricane Center with other models. And, instead of building a new app from scratch for each story, Ben and his colleagues are [developing reusable app templates](https://palewi.re/posts/2024/10/30/data-journalism-delivery/) that easily integrate with their existing tools. The value of these automated pipelines and templates, Ben explains, is in helping data journalists put their energy where it matters most: “All that data work that people had to learn how to do, or had to spend a lot of time doing, was really time they weren't spending focused on what really matters, which is the story and the message.” ## **Innovating in a changing industry** For his back-end data engineering work, Ben’s a self-described “big Python guy,” but he nimbly jumps between thoughts on Python, HTML, node.js, and R’s ggplot2 (and the grammar of graphics more broadly), leaving no doubt that he’s a broad-spectrum polyglot. While he has a suite of go-to tools (“back-end data engineering in Python, front-end with node.js”), he’s also quick to test new technologies, in part to stay at the cutting edge of data journalism in a tenuous time for the industry. “The entire profession of data journalism, which had been built around certain distribution networks and methods, has been totally remade by the internet. I think that that's why so many of us who stick with it — myself included — really embrace technology and innovation. It's a way that you can try to preserve these institutions that have really strong societal value, but need to evolve and adapt to survive in this new economy.” That pressure to keep data journalism thriving in today’s news economy, along with the day-to-day struggle of working with widely different and often quirky data sources, means that Ben and his team are often early adopters of new tech. “In the course of a given year a data journalist will look at dozens or hundreds of different data sets. We’re constantly encountering all the different weird things that can happen with data. Being the nerd caught in the middle of that mucky data processing is something that most people who work in data journalism end up liking. But it’s also why any technology that makes it easier for us is really exciting, and is why we tend to be rabid early adopters of weird new data tools.” Being quick to adopt new tools doesn’t mean Ben is constantly on the lookout for more advanced methods, or more complex tooling. In fact, he seems most fired up about tools that offer a simpler way to do and share data work. For example, he highlights the ease of building with [static site generators](/content/blog/what-is-static-site-generator/index.html) several times in our conversation. "It's been great to see static site deployment really embraced. There are so many good reasons to adopt these simplified deployment routines. Static deployment can have gigantic benefits in terms of reducing financial costs, in terms of sending it out, in terms of how much time it takes to develop and publish, in terms of the risk of being hacked, the long-term maintenance costs…we can just keep going down the list." It’s wisdom gained from experience: > “Building a dynamic website, when you really just need a static site — that’s a great way to learn the hard way.” Ben Welsh To demonstrate, he shares his screen to show us a static dashboard he’s recently built with [Observable Framework](/content/framework/index.html), which acts as both a systems monitoring site for several of his automated data pipelines, and an internal resource library. “It's basically just a simple monitoring site to keep track of our system day-to-day, updated every half hour with the latest data from the data loaders, pulled in from our logs, with all the activity loaded into a SQL database and then fed into this very humble application.” The Reuters Data Wire application, built with Observable Framework, provides updated systems monitoring data, reusable templates, documentation, and other team resources for easier tracking and reuse. Switching to another page, Ben scrolls through several charts while commenting on the simplicity of building and getting value from the app. “The system monitor has very simple charts made in [Observable Plot](/content/plot/index.html) that let us keep tabs on what we’re pushing through day-to-day. You can pick any of the individual systems and drill down to see what’s going on with their heartbeat, or see if there's any that haven’t updated in a while (which would be a red flag). It’s a very simple monitor we had up and out in a day and a half, and it took virtually no work. And then we can add more and do more as we go, right?” ## **Adding and doing more with open source** Ben asks that question — “We can add more and do more as we go, right?” — offhandedly, but it captures an adventurous approach to trying and testing new tools that has permeated the conversation. One way that’s manifested in his work is through open-source software and resources, to which he is a [prolific contributor](https://github.com/palewire/). It’s an effort he downplays at the start (“I just try to learn something new every day…”), but that has resulted in an impressive library of reusable code, software packages, templates, lessons, notebooks, websites, and more. The breadth of tech he’s contributed to is as astonishing as the volume. APIs. Python packages. Observable Framework examples in JavaScript (like this [data loader to generate a PNG from Canvas](https://observablehq.observablehq.cloud/framework-example-loader-canvas-to-png/)). A bot that scrapes and reports data from the U.S. Census…and one that “posts raps by Del Tha Funkee Homosapien to @MISTADOBALINA on Twitter.” According to Ben (who at one point describes himself as a “promiscuous” learner when it comes to software), open source development can be a self-serving practice with a generous outcome. > "Open source is really part of my process of getting unstuck, learning and contributing back to the community, and also helping future me have an easier time. ‘Me’ is probably the number one beneficiary of my open-source software work. To be honest with you, a lot of it is selfish. It's really about making me more productive, happier, and less stressed. For people who wonder why we should do open source, I think that they should consider that they themselves may benefit more than they realize." Ben Welsh Even with so much time spent in the weeds of back-end tech and tooling, he keeps an eye on the bigger picture, frequently returning to his larger purpose: to help improve the stories that journalists can tell with data. “Open-source software allows us to collaborate with people we might think of as competitors on some of that back-end tooling stuff, so that we can compete on the stuff that really makes a difference at the end of the day.” ## **Looking forward and thinking...medium?** So, for someone who’s always looking for the next useful thing to learn, what exciting things are on the horizon? To start, Ben shares a curious and cautiously optimistic view of what AI (“a truly stupendous solution in search of a problem”) can bring to data journalism, from speeding up tedious data processing tasks to making useful analyses possible for more people: “There are recent breakthroughs in AI that are, in the true definition of the word, marvelous. From experiments I've seen, LLMs really seem to have a knack for making some traditional machine learning tasks more accessible to the average person or developer. And, we might be able to have it take on rote or basic tasks when it comes to data pipelining and cleaning. Being able to better democratize a lot of our data work has a lot of power.” However AI ends up impacting data work for journalism, Ben hopes that even the conversations it has spawned will inspire forward progress. > "I've just really loved how the excitement and hype around AI has started a lot of conversations about automation in general, whether it ultimately involves an LLM or not. The interest, the capital, the conversations that have been started in this recent bubble that we're all inflating together — I think these can be harnessed for good and our shared profit if we focus on how we can use automation and structured thinking about information processing." Ben Welsh While discussions around AI often focus on rapidly advancing LLMs trained on massive datasets, Ben is also thinking a bit smaller, sharing shortcomings he sees with how data journalists can visualize and tell stories with “medium data”. “On one end is big data — the gigantic copies of the Internet that fill up Google data centers…so big, you can't even comprehend it. On the other is small data, where you can see everything in a single spreadsheet. But more and more, data journalism projects fall into this middle space that I call ‘medium data.’ There haven't been very good tools for dealing with 10 million, 50 million records for either back-end analyses or, especially, front-end data visualization and publishing. How do I put a hundred thousand points on a map without crashing my user’s browser? What about when they're on a mobile phone with weaker Internet or processing power?” He gives another nod to static deployment as a step in the right direction for medium data, adding, “It’s been really great to see this innovation happening now in the static world and with other new tools that mean we can have all the benefits of static site publishing, while being as ambitious as we can dream of being in terms of how much data we bring to bear. I think that that's something that's still percolating, and it's a really exciting place to be.” As we wrap up the conversation, Ben casually remarks on how our call fits into his day: “Was I too nerdy? In my day-to-day job, I don't get to be too nerdy. This is the only conversation I'll have all day about code. The rest of it’s going to be about news.” Coming from someone who works daily to cut down on time spent coding to make more time for creating and sharing important stories, we think that sounds about right. _A huge thank you to Ben for sharing his work, experience, and expertise with us. Learn more_ [_about Ben_](https://palewi.re/who-is-ben-welsh/) _, and check out his_ [_open-source contributions_](https://github.com/palewire/) _._ _See how Observable can help your team build and host custom, interactive data apps and dashboards at_ [_observablehq.com_](/content/site-root.html) _._ \]\]>https://observablehq.com/blog/why-never-crop-y-axis-part-2-how-to-and-what-to-do-instead https://observablehq.com/blog/why-never-crop-y-axis-part-2-how-to-and-what-to-do-insteadWed, 30 Oct 2024 00:00:00 GMTIt's generally considered a bad idea to start a bar chart’s vertical axis at a value other than 0, and there’s research to back that up (we covered this in [a recent blog post](/content/blog/never-okay-crop-y-axis-except-when-it-is/index.html)). Doing so distorts the comparison between the bars and exaggerates their differences. There is a useful rule of thumb for deciding when to include zero in line charts, but for bar charts it’s really never a good idea to crop the y-axis. In this post, I discuss an alternative to truncating the vertical axis and look at a similar issue in maps and scatterplots. But first, I want to dive a little deeper into the mechanics of cropped axes in visualization grammars, in particular Observable Plot. ## **Cropping axes in a visualization grammar** Visualization grammars can be opinionated about what they support or make easy to accomplish, in particular things that are considered bad practices in visualization. [Observable Plot](/content/plot/index.html) is a good example of a visualization grammar that creates zero-based bars by default, but lets us truncate them if we want to. First, let’s make a regular bar chart from a dataset in a variable called _medals_, which contains medal counts for different countries in a recent Olympic Games. It has two data fields, _nationality_ and _count_. To create the chart, we specify a [barY mark](/content/plot/marks/bar#barY/index.html), map the _nationality_ field to the x axis, and _count_ to the y axis. The code here also specifies a fill color for the bars (steelblue), sorts countries in descending order by medal count, and only shows the first 10. ``` marks: [\ Plot.barY(d3.sort(medals, (d) => -d.count).slice(0, 10), {\ x: "nationality",\ y: "count",\ fill: "steelblue",\ sort: {x: null}\ }),\ Plot.ruleY([0])\ ], ``` This is what the resulting chart looks like: The full code also contains a definition for the x and y axes, though for the x axis, we’re only specifying the axis label. The y axis also defines the _domain_, or range of values in the data set, to extend from 0 to 600. ``` x: { label: "Nationality" }, y: { domain: [0, 600], label: "Medals" } ``` What happens if we change the vertical axis to be cut off, say at 300? ``` y: { domain: [300, 600], label: "Medals" } ``` If we don’t make any other changes, the bars now extend below the x axis, because they still start at 0! They are cut off here by the SVG element the chart sits inside of, though. We can correct that by also setting an additional channel, _y1_, to 300. ``` marks: [\ Plot.barY(d3.sort(medals, (d) => -d.count).slice(0, 10), {\ x: "nationality",\ y1: 300,\ y2: "count",\ fill: "steelblue",\ sort: { x: null }}),\ Plot.ruleY([300])\ ], ``` By default, _y1_ is set to 0, since Plot assumes that bars start at 0. This is a reasonable assumption (and, as we discussed last time, the right choice for bar charts), but it can be overridden. And it lets us create the chart we wanted. So yes, it is possible to crop the y-axis of a bar chart with Plot (and other visualization grammars), but it is of course still a bad idea. Read on below though, for a better alternative! ## **Relative charts instead of cropping the y-axis** One way to avoid the issue of including zero is to show the difference from a reference. This can be a target value, or simply the mean or median of the data. The bars in your chart then show that difference rather than the absolute numbers. This is similar to an index chart, which shows time series data relative to a chosen value (usually a particular reference date). The reference needs to be chosen with care, because whether bars point up or down is very visually salient. Bars pointing down should represent a missed target or some other room for improvement, because they will stand out and demand more attention than ones pointing up. The zero can also be “us” versus “others,” in which case bars pointing up are the competition we should be paying attention to. Take our Olympic medals data from earlier as an example. Let’s say we want to look at medals from the perspective of France, with its 410 medals in these particular Games. We can set our reference at 410, and get this chart: Note how bars pointing up represent countries with more medals than France here, and bars pointing down show countries with fewer. This chart is easy to make in Observable Plot, which allows us not only to specify column names for fields, but also functions. Here, I’ve added a simple function that subtracts 410 from the _count_ value in the data row (Plot passes each row into this function when creating the chart). ``` Plot.barY(d3.sort(medals, (d) => -d.count).slice(0, 10), { x: "nationality", y: d => d.count-410, fill: "steelblue", sort: {x: null} }), ``` Showing relative values means that our bars still start at 0, but since the reference has been shifted, we make better use of the vertical space. ## **Charts other than bar and line charts** So far, we’ve only looked at bar and line charts (the latter in [the previous post](/content/blog/never-okay-crop-y-axis-except-when-it-is/index.html)). These are two of the most common chart types, and they are often used somewhat interchangeably when showing data over time. And yet, they are quite different, especially when it comes to how they depend on the vertical axis. There is a related issue in scatterplots and maps that show a value as the area of a dot or other mark. A value of zero would translate into a dot of size zero, but that would not be different from a missing value. Often, data visualization designers want to still show a mark when the value is zero, and it has to be large enough to be visible and recognizable. That introduces a similar offset problem, where values larger than zero need to be represented by marks that are larger than those showing zero, but the size relationships are no longer correct. The best way to show zero in this case tends to be with a different mark, such as an empty circle if the data is shown with filled circles. This doesn’t entirely solve the problem if the range of values is large, and positive values close to zero have to be shown as well. But it helps avoid the same kinds of confusion as cropping the y-axis on a bar chart. Area charts are something of an in-between of bar and line charts. They are often used instead of line charts to make a chart have more presence, but technically the area under the chart should be meaningful. An area chart of, say, sales over time will have an area that increases with both time and sales numbers. Cutting it off at a value other than zero means that the area is no longer proportional to the sum of sales, but in this case it underrepresents it instead of exaggerating (like it does with a cropped bar chart). An area chart’s vertical axis should therefore start at zero. ## **What about negative values?** All of the above discussion assumes that all our values are positive. That is often the case in business data, for example when looking at sales (in dollars or units), customer numbers at different locations, or time spent on support cases. When the data includes negative values or zero, the vertical axis obviously needs to include 0. But beyond that, one decision to make is whether to simply extend the scale according to the data, or keep it symmetrical around 0. This will depend on what data is being shown, but it can be helpful to see if the data covers the same range on both sides. As an example, here’s the relative bar chart from above, this time from Great Britain’s perspective. The chart on the left is symmetrical around the x axis, the one on the right is not. The left chart is more neutral, while the one on the right emphasizes the bars that point up. Whether this is useful for a particular chart depends on the data being shown and the questions it is meant to answer. It is usually easier to find clear-cut rules of what _not_ to do (such as cropping the axis in bar charts), than come up with simple rules of what to do. ## **Conclusion: it’s complicated!** There is one simple rule here, which is never to truncate the vertical axis in bar charts. Beyond that, things get more complicated and context-dependent. Whether a relative bar chart is possible or meaningful depends on the data and the purpose of the chart. Maps and scatterplots have their own issues when it comes to data including zero (or even negative numbers!). Rules can only take you so far, however. In the end, the visualization designer needs to decide on the right kind of representation of the data. It helps to take a step back and ask, what am I trying to show here, and what do expect the audience to get from this chart? Data visualization is just as much (if not more) art as science. To read more about good chart design, take a look at these related blog posts: - [It’s never okay to crop the y-axis, except when it is](/content/blog/never-okay-crop-y-axis-except-when-it-is/index.html) - [Five techniques to improve chart annotations](/content/blog/five-techniques-to-improve-chart-annotations/index.html) - [Deliver big insights in small spaces](/content/blog/big-insights-small-spaces/index.html) \]\]>https://observablehq.com/blog/ryan-murphy-actblue https://observablehq.com/blog/ryan-murphy-actblueFri, 18 Oct 2024 00:00:00 GMTIn July 2024, as U.S. election news took a dramatic turn, one Observable Notebook surged in popularity: the [ActBlue Ticker Tracker](/content/@rdmurphy/actblue-ticker-tracker/index.html). Created by [Ryan Murphy](https://github.com/rdmurphy), this dashboard accesses and visualizes contributions to ActBlue for “democratic candidates and progressive causes,” as reported by the [ActBlue website](https://secure.actblue.com/). The tracker saw a massive spike in traffic, topping Observable‘s charts with over 950,000 views after Joe Biden’s withdrawal from the 2024 Presidential race and Kamala Harris’s subsequent nomination. ActBlue contributions since September 2020, as visualized in Ryan Murphy's ActBlue Ticker Tracker. We reached out to Ryan, a former data journalist and now a seasoned developer with [The Marshall Project](https://www.themarshallproject.org/), to learn more about the inspiration behind this tool, how it works, and why it continues to capture attention during key political moments. ### **Q: What inspired you to build the ActBlue Ticker Tracker?** **A:** Before The Marshall Project I worked at the Los Angeles Times and the Texas Tribune, and in both newsrooms I specialized in election and campaign finance coverage. Even though I’ve drifted away from that in my current day-to-day work, old habits die hard and I'm always looking for new opportunities to continue exploring those spaces. At the top of the ActBlue website they have this giant big number highlighting the total amount raised from ActBlue contributions. I had always thought it’d be interesting to track it: How does it actually change? What is it actually tracking? Is there any logic behind it? Screenshot of the ActBlue website (from Oct 4, 2024), with the “giant big number” that Ryan’s ActBlue Ticker Tracker has scraped and visualized in an Observable Notebook since September 2020. I went back and revisited the GitHub repo for the ActBlue Ticker Tracker, and the initial commit was on September 18, 2020. I looked back to see why it was _that_ day that I made this and there actually was a reason! It was the passing of Ruth Bader Ginsburg. I wish I had a better snapshot of my brain at that point, but my best guess is that I knew there would be a major fundraising push off of Ginsburg’s passing, and I thought if there’s any time to see if this number would change based on current events, it’s now. So I threw together a chart or two so I could revisit the notebook and see how the ticker was updating. ### **Q: Can you share any “behind the scenes” insights about how you built the scraper to access the ActBlue data?** **A:** The fascinating thing about [the scraper](https://github.com/rdmurphy/actblue-ticker-tracker/blob/main/scrape.sh) is that it’s very minimal. ActBlue surfaces that number in a JavaScript file. The scraper downloads that file, and I strip out anything that’s not the contribution total and then I commit it on GitHub. Then I [run a script powered by a GitHub Action](https://github.com/rdmurphy/actblue-ticker-tracker/blob/main/insert.sh) that checks the git history of changes and builds a CSV based on the diff. So there’s this giant CSV getting a new row appended to it every time the value changes. The script is intentionally as simple as it can be. I wanted to remove the opportunities for me to be clever and break it. _See Ryan’s_ [_ActBlue Ticker Tracker code on GitHub_](https://github.com/rdmurphy/actblue-ticker-tracker) _._ ### **Q: Why did you choose to use Observable for this project?** **A:** Across my career I’ve built my fair share of dashboards and static sites. I could have built this project outside of Observable, but I would have also been building the infrastructure of a house that I didn’t really need to build. I knew I could do it if I took the time but Observable was already there and a good fit for this project. Another benefit is that I can share a notebook and post it to socials (or wherever), where other interested people can see it. With Observable you can quickly build a tool and let people use it. If they want to see how I prepared the data or built the charts they can see the code right there. I’ve built other notebooks where users can drop in their own data. I think that’s always been a really powerful feature! Some of that doesn’t necessarily apply to the ActBlue Ticker Tracker, but it was a big part of why I was drawn to Observable. The GitHub scraper was the higher priority just to get the data pipeline in place. But to have somewhere where I could see in real time the updates coming through was nice. That was also one way we used Observable at the LA Times. I was on the team responsible for tracking COVID in California, and we had an Observable notebook where we could see the status and outputs of all of our scrapers. We were also able to share that with people who didn’t care about the code, and they could see those updates without having to understand our hornet’s nest of Python, AWS, etc. ### **Q: Were you surprised by the notebook’s popularity? Why do you think it continues to draw in viewers?** **A:** Yeah, definitely! I’ve always been surprised that it has continued to be so popular outside of the original political journalist groups I shared it with. The ActBlue Ticker Tracker is a rare opportunity to get a _close_ to real-time view of when money is being raised and often you can guess why. When Ruth Bader Ginsburg passed, it was pretty clear cut why ActBlue contributions had spiked. When there’s a debate or major announcements (like when a new candidate is introduced), it’s pretty easy to see the cause and effect. Again, we do not know who’s getting that money. We cannot say definitively it’s always the Harris campaign raising all this money because many other candidates and political groups also use ActBlue. But you do get this very minor finger on the pulse — it is a brief look behind the curtain that is often very hard to get this early with campaign finance reporting. ### **Q: What surprises, patterns, or challenges have you realized from building the ActBlue Ticker Tracker?** **A:** Really just the fact that people keep looking at it — that’s definitely the most surprising thing to me! I kind of expected it to just fade away. I originally did not have the explanation and caveats at the beginning because I built this primarily for myself and other campaign finance nerds. I assumed most people viewing it had a base level of understanding. But this year after seeing the ActBlue Ticker Tracker again get a lot of attention I felt an obligation to be clearer about what it actually is. For example, the biggest thing that people assume is all of this money is going to the Harris campaign. We actually don’t know! It’s a pretty good guess, but we have no way of knowing because ActBlue is a fundraising vehicle for much more than just the presidential campaign. So this year I made the call that you have to eat your vegetables before you get to see the charts and added a few explanatory paragraphs at the top. I want people to understand that while this is very interesting to look at, there are still so many unknowns. Ultimately the clearest picture will come when ActBlue and the committees file their own campaign finance reports with the FEC. ### **Q: Anything else you want to share about building reports and dashboards with Observable?** **A:** I came up in an era of online data journalism where showing your work was valued and encouraged. We’re often drawing conclusions and making significant statements of fact, and it’s important to give people the chance to see how we came to those conclusions. Observable is wonderful for letting people follow the path you took and see how you reached a conclusion or solved a problem. There are many ways that you can host things on the web and sometimes those will make more sense than Observable. But Observable is great if you want to give people the chance to see your data sources or play with the data themselves (or plug in their own), leave a comment or make a suggestion (I've received a few suggestions on the tracker that were helpful!) or just welcome other people into where your head was at the time. _We’re sincerely grateful to Ryan for sharing his experience and insights with us._ _Learn more about_ [_Ryan_](https://www.themarshallproject.org/staff/ryan-murphy) _and his work with_ [_The Marshall Project_](https://www.themarshallproject.org/) _, “a nonpartisan, nonprofit news organization that seeks to create and sustain a sense of national urgency about the U.S. criminal justice system.” He can also be found on_ [_GitHub_](https://github.com/rdmurphy) _,_ [_Mastodon_](https://social.lol/@ryan) _, and_ [_Bluesky_](https://bsky.app/profile/ryan.fail) _._ _See how Observable can help your team build and host custom, interactive data apps and dashboards at_ [_observablehq.com_](/content/site-root.html) _._ \]\]>https://observablehq.com/blog/never-okay-crop-y-axis-except-when-it-is https://observablehq.com/blog/never-okay-crop-y-axis-except-when-it-isTue, 08 Oct 2024 00:00:00 GMTWhen creating a chart, one of the decisions you have to make is what range of values to show. A common visualization rule says that you’re supposed to start the vertical (Y) axis at 0, at least with bar charts. Check out this pair of example bar charts: Both charts in this image show the same data about the number of medals by country in a past Olympic Games. The left chart, however, starts at zero, while the one on the right cuts off all numbers below 300. It’s easy to see that the chart on the right exaggerates the difference between countries. The top-ranked U.S. has about twice as many medals as Spain (ESP) in tenth place, not 20 times as many as the chart on the right might have you believe. The risk here is that people looking at this chart will miss that it’s cut off, misread the values, and potentially base an important decision on a misunderstanding. This isn’t just a theoretical problem, studies have shown that this is indeed something people reliably get wrong. We’ll talk about one such study below. This has led to the general rule to never start a bar chart at a value other than zero. In fact, many visualization tools and charting libraries make it difficult or even impossible to create bar charts with a non-zero baseline. ## **Theory vs. practice: the line chart** Now this is all good and well with bar charts, since we read bar charts by length. But what about line charts? We supposedly read line charts by position of the points, so they should be immune to having their vertical axis cropped. See for yourself in this chart of Apple’s stock price over about 2 years, starting at 0 on the left and about 90 on the right. These charts clearly look different, the vertical change from one end of the chart to the other is about twice as much in the chart with the cropped axis. While we might read line charts by position, we often care more about the slope of the chart, which is impacted by the range of values on the vertical axis. This is not a new observation. In his famous 1954 book, [_How to Lie with Statistics_](https://archive.org/details/how-to-lie-with-statistics-darrell-huff/mode/2up), Darrell Huff uses a line chart to illustrate what he calls a “gee-whiz graph.” He creates it by taking a line chart that originally starts its vertical axis at 0, cropping it, and then rescaling the chart back to the same size (the person in the background is there to indicate the amount of cropping and distortion). The chart on the left has its vertical axis extended all the way to zero. On the top right, it has been cropped, but is otherwise the same slope. Huff argues that this is a distortion, because it makes it seem as if the numbers increased by 100% over the course of the year. The bottom right chart stretches a very small vertical range across the same vertical area as the original graph. Clearly an exaggeration! This example is interesting because it somewhat goes against Huff’s intended point. The first chart takes up a lot of space, making it impossible to see any pattern like the bottom right chart is showing. The inclusion of the zero doesn’t actually seem to be very meaningful for interpreting the data, or helping to read the chart. But either way, the choice to cut off the axis or include zero makes a big difference. The question is, are people able to understand what they’re looking at? ## **What the research says** In a paper with the great title, [_Truncating the Y-Axis: Threat or Menace_](https://mcorrell.medium.com/truncating-the-y-axis-threat-or-menace-d0bce66d4d08), Michael Correll, Enrico Bertini, and Steve Franconeri report on a set of studies where they tested the effect of cropping the y axis. They look into ways of indicating that the axis was cropped, as well as any differences in how bar and line charts would be affected. In short, they find that there really is no difference between bar and line charts. People misread both kinds of charts, even when there are prominent indicators that the vertical axis has been truncated. In a way, that is what we want: the chart is supposed to work as a graphical representation, and not require us to do mental math with the numbers on the axes. But it also means that we need to be very careful when designing charts and data visualizations, and can’t rely on people being able to spot a cropped axis. This might mean that Huff was right, and all charts really should start at 0 – but the authors of this paper reject that. Instead, they think that it comes down to a judgment call: “ _The designer of the visualization, by selecting a y-axis starting point, has control over the subjective importance of the resulting differences._” ## **If zero is in the neighborhood…** This all might be a little unsatisfactory and not very helpful in practice. There is light at the end of the tunnel, though! Famous statistician and blogger Andrew Gelman has come up with a pithy rule of thumb, “ [if zero is in the neighborhood, invite it in!](https://statmodeling.stat.columbia.edu/2021/12/17/graphing-advice-if-zero-is-in-the-neighborhood-invite-it-in/)” – and has illustrated it with a simple example. What this means is that when your range of values bottoms out close to zero, relative to the range of values, extend it down to zero. This doesn’t compress the chart too much, and helps avoid misunderstandings. If the values are in a range nowhere near zero, leave it out! Going back to the example from _Lying with Statistics_, zero is nowhere near the values being shown, and should therefore not be included. In fact, including zero there compresses the line to the point where it no longer shows useful information. ## **Conclusion** Gelman’s rule is helpful for line charts, and while it does leave some wiggle room, it is quite straightforward and easy to follow. It does doesn't entirely take the decision out of the hands of the data visualization creator, and that’s a good thing. Data visualization is not an exact science, so taste and judgment are still important for good results. Bar charts should always start at zero, however. This is also supported by the _Threat or Menace_ paper I mentioned above, and remains a good, general rule. Beyond bar and line charts, this question is usually less problematic. Area charts, when the area is of interest (rather than just for aesthetics), should start at 0, just like bar charts. The axes on scatterplots are usually scaled purely by the range of the values shown, since these really are read as position (unlike line charts, where slope is often more important). For more on building better charts and dashboards, check out our other blog posts: - [Five underused charts for richer dashboards](/content/blog/five-underused-charts-for-richer-dashboards/index.html) - [Deliver big insights in small spaces](/content/blog/big-insights-small-spaces/index.html) - [Seven ways to design better dashboards](/content/blog/seven-ways-design-better-dashboards/index.html) \]\]>https://observablehq.com/blog/five-techniques-to-improve-chart-annotations https://observablehq.com/blog/five-techniques-to-improve-chart-annotationsThu, 03 Oct 2024 00:00:00 GMTAnnotations help to add context and focus to a chart. They can be static, when the chart designer wants to highlight elements such as outliers or inflection points to tell a story. Chart annotations can also be interactive, with the user driving the exploration with the mouse. Annotations might be bound to the data (like interactive tips showing details on demand), or simply reflect the current mouse position in the coordinate system, giving the user a sense of orientation. We sometimes want to make annotations work across series (in particular for time series), or across facets (e.g. for easier cross-chart comparisons when data are split across small multiples) within a larger [data visualization](/content/learn/what-is-data-visualization/index.html). In this post, we explore five chart annotation patterns that can improve user engagement in data visualizations, and share code to help you build more effective charts in [Observable Plot](/content/plot/index.html). ## Pointer-driven marks “If you make a tooltip or rollover, assume no one will ever see it,” said Archie Tse, Graphics Director at The New York Times, in 2016. Making an interactive tip with a visible default state when the user is not interacting helps inject more editorial guidance into a data visualization — while still allowing the engaged user to drive their own exploration of the details. Starting from a recent example published in the New York Times (a [timeline tracking electoral polls](https://www.nytimes.com/interactive/2024/us/elections/polls-president.html)), we implement a pointer-driven mark that defaults to a static annotation when the user is not interacting with the chart. As soon as they start hovering over the chart, the user seamlessly takes control of the chart annotation and can start exploring on their own, helping drive increased user engagement. Prior to user interaction, a static tip highlights only the most recent polling values. When a user hovers over the data visualization, they take control of the annotation. Learn about the logic in this step-by-step walkthrough on implementing [pointer-driven marks in Plot](/content/@observablehq/plot-pointer-driven-marks/index.html). ## Continuous crosshairs Plot’s [crosshair](/content/plot/interactions/crosshair/index.html) uses the pointer transform to snap to the data. In some cases, you’ll want the interaction to report instead on the current pointer position in the data _space_— to give a sense of orientation, or to understand _where there is no data_. A continuous crosshair thus focuses on the axes in the coordinate system. This also allows a chart to be an input for coordinates ⟨ _x_, _y_⟩. See how to implement [continuous crosshairs in Plot](/content/@observablehq/plot-continuous-crosshair/index.html), and check out how they’re useful when exploring the [diamonds dataset](/content/@observablehq/plot-continuous-crosshair#carat/index.html). ## Cross-facet crosshairs Reacting to the pointer-driven marks notebook described above, [Yuri Vishnevsky](/content/user/@yurivish/index.html) asked: “Can we highlight all ‘related points’ across all facets when hovering on a single point?” The answer, as shown below and implemented in [this notebook](/content/@observablehq/cross-facet-crosshair/index.html), is _yes_. With cross-facet crosshairs, the chart becomes an input for coordinates ⟨ _x_, _y_, _fx_, _fy_⟩. (As an aside, this features an alternative implementation of the continuous crosshair.) See the code to create [cross-facet crosshairs in Plot](/content/@observablehq/cross-facet-crosshair/index.html). ## Staggered axes Sometimes you have a lot of tick labels to display on an axis — too many, in fact, to be readable due to overlap. A classic technique to avoid occlusion is to stagger axis text on alternating rows. But what if a two-row stagger still doesn’t solve your occlusion issues? Here, we take text staggering to the extreme by offsetting labels across 10 rows (below, for a bioinformatics chart visualizing protein sequences) to keep a great number of tick labels separated and readable. For even more clarity, the axis text follows the pointer, highlighting the aligned tick label as a user hovers over the chart series. See the code to create [staggered axes in Plot](/content/@observablehq/plot-staggered-axis/index.html). ## Grouped tips A very common annotation pattern, for charts with multiple series, is to add an interactive tip that simultaneously displays the _y_ values for all the series at a given _x_ position (say, by date). In this example, we use Plot’s render option — which lets you override or extend the default mark’s rendering method — to create a custom tip mark that shows the current value of all series in a stacked area chart. See how we create [grouped tips in Plot](/content/@observablehq/plot-grouped-tips/index.html). ## Build bespoke, interactive charts and data visualizations with Observable Annotations add context and help viewers dive into the data for deeper insights and deliver a more engaging user experience. The five examples above highlight unique and useful chart annotation patterns, implemented in Observable Plot. To start creating custom, interactive charts with Plot, here’s where to start: - Visit the Observable Plot [documentation](/content/plot/index.html) and [gallery](/content/@observablehq/plot-gallery/index.html) - Start exploring and prototyping in [Observable Notebooks](/content/documentation/notebooks/index.html) - Transfer your work into rich, fast [Framework](/content/framework/index.html) dashboards \]\]>https://observablehq.com/blog/big-insights-small-spaces https://observablehq.com/blog/big-insights-small-spacesThu, 26 Sep 2024 00:00:00 GMTData is getting bigger, viewers are increasingly data-literate, and developers — empowered by code-based tools — are creating richer, faster, interactive graphics. At the same time, the spaces in which viewers interact with data visualizations (including on mobile devices, or as part of dense dashboards) are shrinking. When you build with limited screen space in mind, or when you need to embed new content in an existing data app, it can be challenging to design visualizations that retain their richness while still delivering the best user experience. In this post, we share chart types and strategies that help you do more with small spaces, while letting viewers dig deeper into the data. ## **Choosing information-dense chart types for more insightful data visualizations** Some chart types and [data visualizations](/content/learn/what-is-data-visualization/index.html) are purpose-built to give viewers more information in less space. Here are some chart types that pack more data into fewer pixels: ### **Show trends for multiple categories with sparklines** [Sparklines](https://en.wikipedia.org/wiki/Sparkline) — miniature and often minimalistic line charts — are a useful option to show trends for multiple categories in a very small space. Because sparklines commonly minimize or omit axes and labels, they are best suited for cases where you want viewers to focus on major patterns (e.g. increasing or decreasing), and not exact values or minor fluctuations. Their small size and simplicity make sparklines a good candidate for additions to big number boxes, tables, and even in text. For example, sparklines added to the big number boxes below allow a user to see general trends in (mock) product purchases data, alongside highlighted values: And, because they’re so compact, sparklines can even add visual interest and meaning right in text paragraphs: While lines are most commonly used in spark charts, others marks like bars, areas, and cells — really, any other mark you think you can shrink! — are possible. See example code to create [sparkline, sparkbar, and sparkheat charts in Observable Plot](/content/@observablehq/plot-sparklines/index.html). ### **Use horizon charts for trends across groups** Horizon charts transform data into overlapping layered bands to reduce the space needed to visualize trends across groups, without the overlap sometimes seen in [ridgeline charts](/content/@observablehq/plot-ridgeline/index.html) and in a smaller area than a [faceted area or line chart](/content/@observablehq/plot-faceted-areas/index.html). (See our recent post on [underused chart types for richer data apps](/content/blog/five-underused-charts-for-richer-dashboards#horizon-charts-save-space-and-highlight-trends/index.html) to see how horizon charts are constructed.) Horizon charts are a good option when you want to compactly show general trends over time and across multiple categories, as shown for traffic data above. See the code and explore the full version in [this notebook](/content/@observablehq/plot-horizon/index.html). Be aware that there is a tradeoff when presenting horizon charts. Overlapping bands are useful for compactly showing the data while avoiding overlap, but make it difficult for viewers to determine actual values. That makes horizon charts a good option when you want users to focus on general patterns across groups, rather than exact values in the data. ### **Visualize categorical data with treemaps** Treemaps can compactly visualize amounts and proportions by category for hierarchical data. With color to encode discrete categories and labels for some or all segments, treemaps can reveal patterns in data (e.g. “What sectors are the biggest greenhouse gas contributors?”) in a compressed format. For example, the [treemap below](/content/@d3/treemap/2/index.html) visualizes file sizes for the [Flare data visualization library](https://github.com/prefuse/Flare/), revealing contents of the vis (gray), util (brown), and animate (orange) directories as the largest contributors to overall library size. Using a treemap doesn’t guarantee a small chart. While they do capture a lot of data compactly, treemaps can still require a large area if you have a lot of categories (and, depending on whether everything needs a readable label, like in the chart above). But, with fewer groups and flexible aspect ratios, treemaps can help your next data visualization by squeezing a lot of information into a little space. Explore treemaps made with D3, including [cascaded](/content/@d3/cascaded-treemap/index.html) and [nested](/content/@d3/nested-treemap?intent=fork/index.html) variants, and see other compact options for [visualizing hierarchies](/content/@d3/gallery?utm_source=d3js-org&utm_medium=hero&utm_campaign=try-observable#hierarchies/index.html) (like circle packing and dendrograms) in the D3 graph gallery. ### **Using radial and circular charts for more data-dense visualizations** Sometimes, you just need a chart that works well in a little square space. Whether you want to visualize scores by category, show proportions of a whole, or highlight linkages or hierarchies, radial charts can be versatile, effective, _and_ work well in a tight spot. Below are four types of radial charts: tidy trees (hierarchy), sunburst (proportions + hierarchy), radar (value or rank by category), and chord (connections/flows). Radial and circular charts like tidy trees (top left), sunburst (top right), radar (bottom left), and chord diagrams (bottom right) can be useful in a tight spot. Circular charts are quite versatile, with options to visualize continuous values (like in this [radial area chart](/content/@d3/radial-area-chart/2/index.html)), networked data (see the entire [networks section in the D3 graph gallery](/content/@d3/gallery?utm_source=d3js-org&utm_medium=hero&utm_campaign=try-observable#networks/index.html)), and much more. And, they’re often overlooked, perhaps due to strong opinions about the usefulness of the most famous (and [most misunderstood?](https://eagereyes.org/blog/2011/in-defense-of-pie-charts)) of radial charts — the pie chart. ## **Use interactivity to enable users to explore data** Interactivity is useful when — for space reasons, or to limit viewer focus — you _don’t_ want to show everything, everywhere, all at once. When you design an interactive chart, you’re expecting a user to choose the scenario(s) they want to explore. That can help you save valuable app space. Here are a few ways that interactive filtering, tooltips, and zooming can help you build small but mighty graphics. ### **Filtering** Filtering to show just a slice of the overall data based on a user selection (e.g., by clicking a radio button or choosing from a dropdown menu) is perhaps the most common type of interaction in data visualizations. For example, the [hotel bookings dashboard](https://observablehq.observablehq.cloud/framework-example-hotel-bookings/) below gives viewers the option to explore data for all reservations, or for specific subsets by booking type (corporate, groups, travel agent, etc.): (Note: sometimes you do want viewers to directly compare static data slices side-by-side. See the section below on small multiples.) ### **Tooltips** Legends — especially those with a large number of categories — can take up a lot of screen space. They can also take up a lot of brain space, requiring viewers to look back and forth to align series (or groups) of a chart with legend values. In many cases, directly labeling values is preferable. But issues can arise when you want to directly label visualizations _and_ you’re building for small spaces, which can result in crowded or overlapping text. Tooltips allow viewers to see labels and other information on hover, saving space lost to bulky legends and letting users explore values in more detail. For example, the mosaic plot below avoids cramped text by combining static labels on the largest groups with interactive tooltips that reveal values for any segment on hover. (Tooltips or no, [Mosaic plots](/content/blog/five-underused-charts-for-richer-dashboards#mosaic-plots-put-proportions-in-another-dimension/index.html) can be a great option to compactly visualize proportions. Learn more in our [recent post](/content/blog/five-underused-charts-for-richer-dashboards#mosaic-plots-put-proportions-in-another-dimension/index.html).) When legends are too bulky or direct labels too crowded, use interactive tooltips for annotations in limited space. ### **Zooming** Here, zooming means expanding a smaller or collapsed part of a chart to enable deeper data exploration. Zooming makes it possible to show an overview chart, while letting a user dig into more detail when they want to, and without expanding the total chart area. Panning is a type of interactive zoom that allows a user to choose the range of values visible in a chart. Panning is particularly useful when you have a large number of dense data points, or frequent values recorded over time, because it allows a viewer to see the whole picture with the option toinvestigate specific regions or periods of interest more closely. For example, this [zoomable area chart](/content/@d3/zoomable-area-chart/index.html) allows a user to expand or contract the x-axis scale, and pan across time: Panning and zooming help viewers to expand the range and resolution of data in a chart, without expanding the total chart size. To visualize hierarchical data in limited space, you can allow a different flavor of zooming — one that allows viewers to dive deeper into nested data. For example, the [zoomable icicle chart](/content/@d3/zoomable-icicle/index.html) below shows the top three levels in a hierarchical dataset. Clicking on an area zooms to only show data within that level, letting a viewer increase or decrease what subset of the total data remains visible. Other options that let viewers dig into hierarchical data include [zoomable circle packing](/content/@d3/zoomable-circle-packing/index.html), [zoomable sunburst charts](/content/@d3/zoomable-sunburst/index.html), and [collapsible trees](/content/@d3/collapsible-tree/index.html). ## **Designing data visualizations for the small screen** ### **Small multiples** Interactivity gives users one way to compare scenarios (e.g. visit our Framework examples to t [oggle between hotel booking categories](https://observablehq.observablehq.cloud/framework-example-hotel-bookings/) or [see shifts in U.S. electricity demand over time](https://observablehq.observablehq.cloud/framework-example-eia/)). But in some cases, viewers benefit from seeing different scenarios side-by-side and on screen at the same time. Small multiples are a compact way to visualize data for different categories or times using faceting. For example, the [radar charts of car characteristics](/content/@observablehq/plot-radar-chart-faceted/index.html) below packs a lot of information into a small area, while allowing viewers to directly compare car types. Small multiples, like in the [faceted radar chart of car characteristics](/content/@observablehq/plot-radar-chart-faceted/index.html) above, allow for direct comparisons (e.g. by category or time) and can pack a lot of information into limited space. And, you’re not limited by chart or faceting variable type! For example, the small multiples below bin a quantitative variable (year) to visualize new store locations by decade, using Observable Plot’s [geo mark](/content/plot/marks/geo/index.html): ### **Using responsive charts and grids** Adding to the challenge of designing visuals for small areas is variability: individual charts and entire displays need to adjust for different screen and window sizes. When a user narrows a browser window or opens your app on their phone, visuals should smoothly adapt while maintaining chart clarity and overall app usability. (See our recent post on [how Observable makes building data apps faster and easier](/content/blog/observable-makes-building-data-apps-faster-and-easier#responsive-layout-and-built-in-components/index.html) to learn more.) That’s why we designed [Observable Framework](/content/blog/observable-makes-building-data-apps-faster-and-easier#responsive-layout-and-built-in-components/index.html)’s responsive grid class and resize helper to make it easy to build flexible data apps and dashboards that adapt to any screen. The [resize](/content/framework/javascript#resize-render/index.html) helper function automatically re-renders charts to div dimensions, and our [grid class](/content/framework/markdown#grids/index.html) is automatically responsive, adjusting column numbers to best fit a user’s screen. Below, notice how chart dimensions are initially updated, then the content is rearranged into fewer columns once a breakpoint for better viewing on a narrow screen: ## **Building visual displays of data that fit any screen** Want to try making these small but mighty data visualizations today? Sign up for Observable to explore examples or fork from thousands of public Observable Notebooks (including those linked above), then customize with your own data. Learn more about adding interactivity to your charts and dashboards with these [tips for better chart annotations](/content/blog/five-techniques-to-improve-chart-annotations/index.html). - Sign up at [observablehq.com](/content/site-root.html) - Explore and fork from thousands of charts made in [D3](/content/@d3/gallery?utm_source=d3js-org&utm_medium=hero&utm_campaign=try-observable/index.html), [Plot](/content/@observablehq/plot-gallery/index.html), or [created by the Observable community](/content/trending/index.html) - Follow along with our [Getting started guide](/content/framework/getting-started/index.html) to create your first Framework data app \]\]>https://observablehq.com/blog/how-build-successful-user-facing-data-apps https://observablehq.com/blog/how-build-successful-user-facing-data-appsWed, 18 Sep 2024 00:00:00 GMTBuilding good dashboards for internal users is not easy, but the challenges and pitfalls [are generally well understood](/content/blog/seven-ways-design-better-dashboards/index.html). Creating data apps for external users is an entirely different matter, with its own set of considerations. User-facing analytics are data apps and dashboards that are made for users external to your organization, usually your customers, clients, or end-users. Perhaps you provide services and want to show your customers how they are performing. Perhaps you’re leasing or selling devices that collect data about their performance and utilization and want your customers to know how they’re performing. Or maybe [you’re creating and publishing rich, interactive maps and graphics that highlight local impacts of climate change](/content/customer-stories/climate-central/index.html). In all these cases, you’re dealing with a broad range of users, who are also usually less familiar with the data being shown. At the same time, you also know less about them, and often can’t easily get feedback from them. To build useful data apps for these users, it’s important to put yourself in their shoes and try to envision the kinds of questions they might have, and the decisions they might need to make. External users are more challenging, because they work in a different context, so need to be reminded that they’re looking at a data app coming from you. That makes branding and design even more important to ensure they understand what they’re looking at. They’re also less forgiving about a slow dashboard than your internal users. User-facing data apps need to enable quick decisions even more than internal ones, and at the same time give people ways of digging deeper if necessary. It’s also a good idea to include more context and pointers to other data than you might have for an internal dashboard. And they need to be snappy, or risk losing your users/customers. Here are a few tips and guidelines for building successful and delightful data apps for external users. ## **Why building data apps and dashboards for external users is more challenging** External users — your customers, clients, or end-users — work in a different context and typically are less familiar with the data being presented, so they need to be reminded that they’re looking at a data app coming from you. That makes branding, design, and visual identity even more important to ensure they understand what they’re looking at. External users are also less forgiving about a slow dashboard than your internal users, so prioritizing the user experience is essential when designing effective dashboards. User-facing data apps need to enable quick decisions and fast insights, and at the same time give people ways of digging deeper if necessary. It’s also a good idea to include more context and pointers to other data than you might have for an internal dashboard. And they need to be snappy, or risk losing your users/customers. In this guide, we’ll cover how to build effective, user-facing data apps and dashboards that are fast, insightful, and easy to navigate. ## Building data apps for quick insights and decisions: the overview dashboard External users, more even than internal ones, will use your data app for a quick check first: _Is everything as expected? Is any action needed?_ Your data app needs to be designed for this task. There are several ways to do this. One is to help users by phrasing headings as questions. _How many users did we have today? How many gizmos did we sell this quarter? How are sales tracking relative to our quarterly goal?_ Etc. Each of these needs to be paired with a relevant data visualization or number, of course. Another is creating a visual hierarchy, so users can find key information quickly without getting overwhelmed. Structuring a dashboard using big numbers at the top (we discussed these and other [strategies for building better dashboards](/content/blog/seven-ways-design-better-dashboards/index.html) recently) gives users the top-line information quickly without having to hunt for it. Other views need to be designed so they don’t compete with these numbers, but connect with them. [Careful use of color](/content/blog/crafting-data-colors/index.html) can help with this, as can using section headings and spacing that group logical units of information. A quick “TL;DR” or executive summary-style section at the top can also help to assure users that metrics are in check, or point them to specific ones that might need attention. This can be generated by your analysis code, and incorporate not only numbers from the overview, but also links to the more detailed views discussed in the next section. Crucially, quick insights mean limiting the amount of information being shown.  An external-facing data app shouldn’t go into too much detail, but keep things high-level with the option to get more information. It is tempting to add more and more information to a dashboard, but at the risk of overloading it and making it less useful for its purpose. ## **Dig deeper: creating data apps for more detailed user-facing analytics** Depending on your use case and users, you might want to build in ways to allow your users to dig deeper. This won’t be necessary for all use cases, but it can be incredibly helpful for your users when there is a problem they need to diagnose. Splitting off detailed analytics views can also help to keep the scope of the overview dashboard more manageable, and keep it focused on the essential information. One good way of doing this is with multiple pages that are linked from the main page of the app. If a user has a question about the data shown on a particular chart, the link to more details should be right there as well. [Using code to build charts](/content/blog/three-ways-build-better-dashboards-data-apps-with-code/index.html) helps to create more and less detailed versions of the same view, too. You can reuse most of the code, with options to control the level of aggregation, which additional data columns to show, etc. Not only will the visualizations be more detailed on these pages, there will also be more controls. This includes filters, ways to add more data columns, options to turn models off and on, and even ways to control models. If you’re building your dashboards with code (as you should!), you can run models right in the user’s browser, which means giving them direct and fine-grained control over them. Models can help users understand what the data is telling them, and play with what-if scenarios. Even with detail views, there are always going to be questions your data app won’t be able to answer. For these, you might want to provide ways to export the data so users can do their own analyses in tools like Excel. A common data format like CSV is ideal for this, because despite its simplicity and flaws, it is readable by virtually any [data analysis](/content/learn/what-is-data-analysis/index.html) tools or [data visualization](/content/learn/what-is-data-visualization/index.html) software. ## **Help people understand: including context in user-facing data apps** External users are usually less familiar with the data and terminology being used in a data app than internal ones. They need more guidance and context to be able to understand, and not misunderstand, what’s being shown. Ways to help users include making sure that all charts are properly labeled, including units on axes, time periods, legends for color encodings, etc. Chart and section titles should also be clear about what is being shown, and additional text can be added to help users interpret what they are seeing. In addition, if there is documentation about the metrics shown, the specific charts, models, etc., link it directly from the app. It is easy to assume that your users will understand what they are looking at, but there are many reasons why they might not: perhaps they aren’t familiar with the app yet, haven’t used it in a while, or are only starting to work with data in their role. Finally, when all else fails, users need to be able to reach a human to ask questions. Including a way to reach support or the developer of the app should be included in the app, and be easy to find. ## **Situate your end users: using branding and design in data apps** Your users might be using many data apps in their work. Make sure they know which one they’re looking at! This includes both branding and thoughtful design. Branding is all about your visual identity: logos, colors, typefaces, etc., that identify your company or product. You want your users to recognize your data app at a glance when they flip through their browser tabs. Good design, in addition to the visual hierarchy mentioned earlier, also means making your data app look great so your users will actually want to use it. A well-designed app is more enjoyable to use, which translates into more of your users’ mental energy spent on their task than wondering about your poor design choices. For both branding and visual design, [Observable Framework makes it easy to create themes](/content/framework/themes/index.html) that can be used across many different data apps and pages. They can be imported just like code, and any changes will automatically propagate through all your data apps for consistency. An important consideration here is also accessibility. External-facing data apps, even more than internal ones, will be used by a wide variety of people with different abilities. This is not only a good idea to make sure all your users can actually use your app, but you might actually be legally required to follow accessibility guidelines. Observable Framework’s use of web standards allow you to quickly incorporate all the assistive technologies that are built into today’s web browsers and operating systems. ## **Speed matters in dashboard design: optimizing load times** Quick insights not only require thoughtful design, but also a data app that loads quickly. Traditional business intelligence (BI) tools load their data when the page loads, which can take a long time. Outside users, even more than internal ones, will lose focus and move on if they have to wait. Since they’re also paying for your services, they will also be less forgiving, and more vocal, about their dissatisfaction than internal users would. Making your dashboard load fast isn’t a luxury, it profoundly affects how people use, perceive, and engage with your work. Users come to your dashboard with a question or the need to check in on their data. Having to wait for the data to load means they might get distracted and move on to other things in the meantime. This makes their work less efficient, and reflects badly not just on your dashboard, but the entire image they have of your products and services. Your databases might be under load, or there might be a lot of data processing happening behind the scenes. Your users won’t know that or, frankly, care. What they see is that you’re making them wait for your data app to load. This is why Observable Framework puts speed first. As a [static site generator](/content/blog/what-is-static-site-generator/index.html), it runs data loaders offline (on a schedule, see the next point) to generate “baked” data packages. These load instantly when your dashboard or data app loads, which makes for a profoundly different experience for the user. It also gives developers more freedom to perform complex queries, data processing, and modeling, since they’re not constrained by load times. ## **Keep your data fresh: scheduled builds** The data in your app needs to be as fresh as makes sense for your application — but beware of [the fallacy of up-to-the-second data](/content/blog/align-dashboards-with-scales-of-business-decisions/index.html). While it might seem that data needs to be as current as possible, data about incomplete days, weeks, or months can be a distraction. If your data shows sales, for example, your users might wonder why they’ve been dipping when in reality they’re looking at data that only contains parts of the last week, month, or quarter. This can lead to misunderstandings and bad decisions. A well-considered update schedule means that the data shown is always meaningful, without the noise, distraction, and cognitive load of incomplete data. In addition, any dashboard or data app should always show when the data was updated, so users can understand what they are looking at. [Observable Framework’s data loaders](/content/framework/data-loaders/index.html) can be run at whatever interval makes sense for your application. Cloud hosting providers typically let you schedule builds using cron jobs or similar. This takes care of running the jobs, but it is important to keep monitoring these scheduled builds for errors to ensure that your data always stays fresh. ## **Where to go next: building successful data apps and dashboards** User-facing data apps can provide a lot of value to your users, and make your products and services much more accessible and valuable to them. Building good ones is challenging, but also very rewarding. Many of the considerations for external-facing data apps also apply internally, but some of them are unique. Branding and visual design, for example, are less crucial for internal users (though they will also appreciate a clean, good-looking dashboard). External users also need more context, and being able to quickly find information is even more important to them if they’re more casual users (in terms of time spent, not importance) of your data. Some of the other items are also important for internal users as well, of course, such as speed, the need to carefully consider how and when to update data, or the visual hierarchy of the information being shown. But since you’re likely to know more about your internal users, you can tailor the information to their needs much more directly, and more specifically, than you can for external users. To explore further, check out these materials: - [A blog post on why building data apps with code](/content/blog/three-ways-build-better-dashboards-data-apps-with-code/index.html) gives you more flexibility and power, especially when you’re building user-facing dashboards - [More on data loaders](/content/blog/data-loaders-for-the-win/index.html), and how they make for snappy data apps, no matter how complex your data processing \]\]>https://observablehq.com/blog/announcing-observable-cloud https://observablehq.com/blog/announcing-observable-cloudTue, 10 Sep 2024 00:00:00 GMT **Update:** As of April 15, 2025, Observable Cloud has been deprecated. Existing instances will continue to function, but new instances cannot be created. _“Light everywhere… Like the whole thing was built of light. I wish you could see what I see.”_ So little of the data around us makes it to our eyes to inform our understanding and help us make decisions. At Observable, we want to fix that. We want you to see the data. That’s why we’re reimagining how data apps are built. We believe there exist two main problems with the status quo: 1. An _architectural_ problem where it’s hard to get rich data to the screen performantly 2. An _organizational_ problem where data teams aren’t empowered to move quickly To address the architectural problem (or [“last mile” problem](https://www.youtube.com/watch?v=n5gFBQTClxc)), we’re betting on the [“baked data” architecture](https://simonwillison.net/2021/Jul/28/baked-data/). In this approach, data is precomputed at build time rather than being queried on-demand when a user views the app; these static snapshots of data load instantly rather than forcing users to wait for expensive database queries to finish. We believe this is the right trade-off because storage is cheap while user time and attention are highly valuable. Plus, faster dashboards are not only more enjoyable to use, they can show [orders of magnitude more data](https://observablehq.observablehq.cloud/framework-example-api/) and afford deeper insights. The baked data approach is reified in our open-source framework for building data apps, [Observable Framework](/content/framework/index.html). Baked data also helps with the organizational problem: by tightly coupling data loading with display, data teams can use their preferred programming languages even while building web apps, easily integrate messy data from disparate sources and APIs, and more quickly iterate on designs. But we want to go even further to address the organizational problem: we want to help data teams deliver better data products directly to end users. So today we’re launching **Observable Cloud**, our seamlessly integrated development and production platform for Framework apps. Observable Cloud lets your data teams ship faster, safely, and deliver best-in-class experiences to your users. The core strength of Observable Cloud is that we’re focused exclusively on being the best place to develop and host Observable Framework apps — and only Framework apps. This specialization offers unique benefits to both users and developers. ### Better performance Observable Cloud understands how Framework apps are structured and optimizes serving accordingly. Framework uses content-hashed asset paths to guarantee that the path of an asset will change whenever its content changes. Assets include data files, JavaScript modules, self-hosted libraries from npm, stylesheets, WebAssembly bundles, etc. The beauty of content-hashed assets is two-fold: First, Observable Cloud can serve assets with [immutable, long-lived caching](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control#immutable). We allow browsers to cache assets for a long time and skip revalidation since we know asset contents won’t ever change. For example, say your app uses observable.png as its icon. In HTML, you’d say: `` When Framework builds the app, it rewrites the above href attribute to apply a content hash (`1af93621`) and moves the asset to a serving directory (`_file`) dedicated for such assets. `` If you ever change your icon, the content hash will change, even if the asset is still called observable.png. So the next time you deploy your app, it might automatically change to this: `` (Content hashes are more complicated for JavaScript modules, as Framework must further consider transitive imports: even if a module itself doesn’t change, its hash must change if a module it imports changes!) Second, content-hashed paths allow atomic deploys of pages. When you deploy a Framework app, users never see a [“chimerical”](https://en.wikipedia.org/wiki/Chimera_(mythology)) version of a page where assets from the old and new versions are commingled, potentially leading to errors. When the app is built, unchanged assets have the same name as before and remain cached, while changed assets have a new name — so whatever version of a page a user loads, the linked assets are always consistent. (This approach will also make it easier for Observable Cloud to serve multiple versions of the app simultaneously, say for branch previews and instant rollbacks; stay tuned!) ### Effortless continuous deployment Observable Cloud is designed to help you ship quickly without tedious configuration. Instead of crafting a custom action in your preferred CI provider, you can simply connect Observable to your source control repository on GitHub and immediately turn on continuous deployment. And thanks to GitHub’s commit status API, you’ll see the status of Observable deploys directly on GitHub. To keep your app up-to-date, you can further add an `observablehq.cloud.yaml` file to enable scheduled builds at your preferred cadence. You can choose whether to run builds on the Observable Cloud or your own private, on-premises environment. ### Effortless secure sharing Data apps, by nature, tend to be private — you only want to share them with your intended audience. Observable Cloud makes it easy to share private data apps seamlessly and securely, either by inviting viewers to your Observable workspace or by using [secret link sharing](/content/documentation/projects/secret-links/index.html). Enterprise customers can further use private cloud serving for network-based access control, and integrate with an external identity provider via OpenID Connect (OIDC). ### Integrated analytics Observable Cloud provides first-party analytics for Framework apps so you can see how your app performs and [fight dashboard rot](/content/blog/five-strategies-fight-dashboard-rot/index.html). Unlike generic analytics, Observable analytics understands the structure of your app — it knows which pages exist. It even knows which pages were deleted in more recent versions. We designed our analytics to give data teams the information they need at-a-glance, such as which pages are popular and which aren’t, and who’s visiting them regularly so you can ask stakeholders for feedback. You can read more about the design of our analytics [on our blog](/content/blog/how-to-track-dashboards-enhanced-analytics/index.html). * * * Reimagining how data apps are built and delivered is — as you might guess — a big undertaking. While the launch of Observable Cloud today realizes our vision of an end-to-end solution from ideation to production (and back again), we’re still just getting started. Try Observable Cloud today and let us know what you think! \]\]>https://observablehq.com/blog/framework-essentials-for-data-teams https://observablehq.com/blog/framework-essentials-for-data-teamsThu, 05 Sep 2024 00:00:00 GMTFast, rich, interactive graphics are the stars of data apps built with Observable Framework. Just behind the scenes, however, are a number of essential assistants that get them to the red carpet. Supporting services, software, and features (like Node.js, npm, data loaders, and the Observable standard library) help to make Framework data apps fast and flexible. Experienced web developers might already have deep knowledge of some of the supporting software, services, and features (like Node.js, npm, static site architecture, and standard output) that Framework relies on. But for data teams and analysts who don’t spend as much time working in JavaScript, they may be less familiar. Whether you’re a data scientist building your own data app with Framework, or part of a data team working with JavaScript developers to create [data visualizations](/content/learn/what-is-data-visualization/index.html) together, it’s helpful to know the basics of what’s going on behind the curtain. Here, we introduce eight essential tools that data teams should be familiar with when building data apps with Framework. ## **Node.js: The JavaScript runtime environment for data apps** To execute JavaScript code — which is the basis of Framework data apps — you need a runtime environment. Browsers already have a built-in JavaScript runtime. But, when you’re working locally and outside of the browser (where you develop Framework data apps), that may not be available by default. [Node.js](http://node.js/) is a "free, open-source cross-platform JavaScript runtime environment." With Node.js installed you can run JavaScript anywhere, including locally, outside of the browser. To learn how to install Node.js, you can find the official docs [here](https://nodejs.org/en/download/package-manager). [Observable Framework](/content/framework/index.html) — our open-source tool for building custom data apps — is a Node.js application. You can install Framework from npm (see below) to develop, preview, and build your data apps locally. That means that Framework projects integrate nicely with your existing file-based data workflows, and you can develop data apps using your favorite code editor. ## **npm: Access millions of JavaScript packages** [npm](https://www.npmjs.com/) is a popular package manager for Node. It is, first and foremost, a software registry — the go-to place from which developers install over 2 million JavaScript packages. Are you familiar with software repositories like [PyPI](https://pypi.org/) (for Python packages) or [CRAN](https://cran.r-project.org/) (for R users)? This is the JavaScript equivalent. Observable Framework is [published on npm](https://www.npmjs.com/package/@observablehq/framework). To create a new data app and ensure you’re using the latest version of Framework, run: ``` npx "@observablehq/framework@latest" create ``` [npx](https://docs.npmjs.com/cli/v8/commands/npx) allows execution of npm packages without explicitly adding them as a dependency. (Note: Node.js installs both npx and npm utilities by default, so you won’t have to install these manually if you already have access to Node.) And, npm isn’t _just_ the world’s biggest grocery store for reusable JavaScript software. It’s also a command line interface that helps you execute Framework commands, for example to build and deploy your data app. **Note:** You can also install Framework, and execute Framework commands, using [yarn](https://yarnpkg.com/). Like npm, yarn is a package manager for JavaScript projects. ## **Static site generators: Deliver data apps that load instantly** Static site generators (sometimes abbreviated as SSGs) build all the files and code needed for a website or data app. The site is then served from these generated, static files. Data teams might be familiar with static site generators like [Jekyll](https://jekyllrb.com/) and [Hugo](https://gohugo.io/), which are commonly used in the data science space for websites, blogs, and static pages. Framework is the equivalent, with a focus on creating best-in-class data apps. When you build with static site generators, time consuming queries and data processing (e.g. [data wrangling](/content/learn/what-is-data-wrangling/index.html) and modeling) are executed before the required page contents are packaged up and sent to the user’s browser. That results in faster page loads, and better app performance. There’s a common misconception that a _static_ site generator can only create non-interactive pages. But, to reference a classic film: > You keep using that word. I do not think it means what you think it means. Inigo Montoya The Princess Bride In reality, static site generators — like Observable Framework — are key to building faster interactive data apps, since charts don’t require new computation each time a user selects a different radio button or changes a slider value. When you build with Framework, you can precompute data for all scenarios you want a user to explore (e.g., using a [data loader](/content/framework/loaders/index.html)). That means users will still be able to interact with your visuals, and — because your app doesn’t re-run queries or analyses from scratch every time a user changes a selection — the interaction is faster, and reduces loads on shared databases. Learn more about static site generators and why they’re great for fast, interactive data apps in our [recent blog post](/content/blog/what-is-static-site-generator/index.html) (also available in [video format](https://www.youtube.com/watch?v=FQL2J7809a8)). **Note:** building a static site doesn’t mean you _can’t_ include JavaScript code that accesses data from an API or database, or that performs client-side computations. But, beware that doing so means your viewers will have to wait until those processes are complete for their page to load, which can be slow. Which brings us to data loaders! ## **Data loaders: Precompute data for fast, interactive data visualizations** Let’s say you want to include a rich, interactive chart in your data app, which visualizes a large number of records that are returned by an involved SQL query. If all data processing happens right when a viewer opens your app on their device, they’ll have to wait for the query to run, and the data to load in their browser, before the chart is rendered. That can be slow. Slow data apps that make users wait are more likely to [rot](/content/blog/five-strategies-fight-dashboard-rot/index.html). Data loaders are the antidote. Data loaders are programs that you can write in any language (Python, R, JavaScript, SQL, or anything else) to generate data snapshots at build time. With data loaders, you can do all of your data pre-processing, analysis, and modeling behind the scenes to minimize what gets sent to the client. The result is less data sent to the client, and less computation required when a user opens your data app — and that means faster pages and happier viewers, who are more likely to return. See the [Framework docs](/content/framework/loaders/index.html), our recent [blog post](/content/blog/data-loaders-for-the-win/index.html) and [video](https://www.youtube.com/watch?v=Q6lQRUTZpnU), and the ever-expanding collection of [data loader examples](/content/framework/loaders/index.html) to learn more about how data loaders work, why they’re awesome, and how to make your own. And, check out our new [page loaders](/content/framework/page-loaders/index.html) in Framework. Whereas data loaders precompute data snapshots for better performance, page loaders “pre-bake” dynamically generated pages for even faster page loads! ## **Standard output: Streamlining data integrations for data apps** To work with Framework’s routing, data loaders must send their output to standard output. What is that, and why is it so important when building your data apps? Standard output is one of the [standard streams](https://en.wikipedia.org/wiki/Standard_streams) in Unix and POSIX systems, and is the expected place where programs write data _to_. What happens when you write a data loader, and where does standard output fit in? If the file generated by a data loader is referenced in a Markdown page, Framework’s routing will first check to see if the output lives as a static file at the designated path. If it does _not_, then it will check the cache — that’s where outputs of data loaders are stored — to (1) see if it exists there, and (2) if the timestamp of the cached file is older than that for its parent data loader. If the data loader has been updated more recently than its last cached output, the data loader is re-run. The data loader output is read **from the standard output stream**, and (thanks to Framework’s routing) is stored in the cache where it is accessed by the Markdown file. The sequence of steps that determine if a data loader is re-run (and the output sent to standard output) when a file is referenced in a Markdown page using FileAttachment. ## **Automatic reactivity: Develop data apps more efficiently** If you’ve ever used an Excel spreadsheet where a change to one cell value is propagated instantly to all other cells that depend on it, then you’ve experienced automatic reactivity. When working in Framework ( [and Observable Notebooks](/content/blog/from-data-exploration-to-data-apps-with-observable/index.html), for that matter), [automatic reactivity is supported _in your code_](/content/framework/reactivity/index.html). Whenever you update a piece of code, any downstream code that depends on it is automatically re-executed. With automatic reactivity, when code is updated, all downstream code depending on it is automatically re-executed, regardless of their relative order. Here, a change to the final line of code in a Markdown file triggers the chart code above it to re-run. This has a number of advantages for your development experience and page performance. For example, you can work more flexibly, since you can organize your code however you want, not just from top to bottom. Since changes propagate throughout all of your code, you don't have to worry about manually re-running pieces to ensure you're seeing correctly updated outcomes. And, because only downstream code is re-executed, pages are more performant (compared to those requiring a full-page reload any time someone clicks on  a radio button). Learn more about how reactivity works, and why it’s so useful, in our [Framework documentation](/content/framework/reactivity/index.html). ## **Observable standard library: Rich and expressive data visualization components** When building data apps with Framework, you can [import any JavaScript libraries](/content/framework/imports/index.html) you want for [data analysis](/content/learn/what-is-data-analysis/index.html) and [data visualization](/content/learn/what-is-data-visualization/index.html), including from npm, local paths, and remote URLs. But some libraries are so useful, and so commonly used, that it makes sense to have them available in Framework by default. Meet the Observable standard library — a collection of JavaScript libraries available out-of-the-box when you build data apps with Framework. That includes powerful libraries for creating interactive data visualizations (like [D3](/content/framework/lib/d3/index.html), [Observable Plot](/content/framework/lib/plot/index.html), [Leaflet](/content/framework/lib/leaflet/index.html), [Mosaic vgplot](/content/framework/lib/mosaic/index.html), [deck.gl](/content/framework/lib/deckgl/index.html), and [Observable Inputs](/content/framework/lib/inputs/index.html)) and analysis (e.g. [DuckDB](/content/framework/lib/duckdb/index.html) and [Arquero](/content/framework/lib/arquero/index.html)). Some of the libraries available in Framework out of the box as part of the Observable standard library. What does that mean for you? With Node.js and npm installed, you can create a new Framework data app and immediately start building interactive data visualizations. See the [Libraries section of our Framework page](/content/framework/index.html) to learn what else is at your fingertips, and learn how the Framework standard library differs from that of [Observable Notebooks](https://observablehq.observablehq.cloud/framework/convert#standard-library). ## **Observable Plot: Create interactive data visualizations with concise code** JavaScript is the premier language for building custom, interactive charts and graphics. It’s also _not_ a language that most data teams use for their day-to-day work — that’s more likely to be a language like Python, SQL, or R. With data loaders on the back end, Framework lets you wrangle and analyze data in any language. But when it comes to creating interactive graphics in Framework data apps, those are built in JavaScript. There are _many_ JavaScript libraries for data visualization. But the one that we recommend for data teams taking the leap into Observable is [Observable Plot](/content/plot/index.html). Plot is an open-source library for building exploratory data visualizations [layer-by-layer with marks and scales](/content/plot/what-is-plot/index.html), based on the [grammar of graphics](https://vita.had.co.nz/papers/layered-grammar.html). For data teams already building charts with other grammar of graphics-based tools like [ggplot2](https://ggplot2.tidyverse.org/) (for R users) and [plotnine](https://plotnine.org/) (for Python users), Plot will feel familiar. Initialize a plot, add marks that represent your data, and map variable values to chart elements like mark size, color, opacity, and more to make increasingly expressive graphics. And, because it’s built on D3, you can extend Plot charts with D3-supported animation or any other bespoke customization. Explore examples and reusable code for hundreds of charts in the [Observable Plot gallery](/content/@observablehq/plot-gallery/index.html). To learn more, read our blog post on [building Framework dashboards using Plot](/content/blog/how-to-build-dashboards-observable-framework-plot/index.html). ## **Conclusion: Start building with Observable Framework today** By understanding the essential tools for working with Observable Framework, analysts and data teams will be better equipped to more efficiently build fast, interactive data apps. Node.js and npm let you work locally to build with Framework (in files, using your favorite code editor!) while using any JavaScript library you like to build the best data visualizations. Data loaders and our static site architecture let you precompute data snapshots, and pre-package page contents, to give viewers the fastest pages and best user experience. The Observable standard library makes powerful tools for data visualization and analysis available by default, including Observable Plot — our go-to library for quickly exploring data and prototyping new charts. With some essentials under your belt, you're ready to start building with Framework! Check out our resources to dive in: - Follow along with our [Getting started guide](/content/framework/getting-started/index.html) to create and deploy your first Framework data app - Check out the [Framework documentation](/content/framework/index.html) - Read our [recent blog](/content/blog/observable-makes-building-data-apps-faster-and-easier/index.html) on how Framework helps you create better dashboards, faster - Learn [why data loaders are a win](/content/blog/data-loaders-for-the-win/index.html) for data teams, app performance, and decision makers - Visit the [Observable Plot](/content/plot/index.html) documentation and [gallery](/content/@observablehq/plot-gallery/index.html) \]\]>https://observablehq.com/blog/seven-ways-design-better-dashboards https://observablehq.com/blog/seven-ways-design-better-dashboardsWed, 28 Aug 2024 00:00:00 GMTSo you’re building a dashboard, but are wondering what to build or how to make it even better. Perhaps you’re not sure about how to lay things out, which charts to include, or which pitfalls to avoid. This post offers seven ideas and guidelines to help you make the right decisions to build better dashboards. A good dashboard serves multiple purposes. First, it gives a quick overview of the key numbers at a glance. The goal is to make it easy to see if things are as expected or if action needs to be taken. If things are going well, the viewer might just move on and not even look at the rest of the dashboard – that’s a good thing, you want that! But when things aren’t as expected, your users need to be able to dig deeper. That is where additional views and filters come in to allow them to answer questions. The dashboard needs to be organized in a way that makes it easy to get around, even when a lot of information is being shown. It needs to be powerful without being overwhelming. The rest of this post will discuss visual hierarchy, the use of big numbers, coordinated views, and interaction, and how to use colors. This all bookended by broader points about asking the right questions upfront, and keeping an eye on usage to make sure your dashboard is meeting your users’ needs. By following these tips, you’ll be better equipped to design dashboards that are not only visually appealing but also user-friendly, informative, and effective. ## Asking the right questions to design the right dashboard The first step in building a dashboard is to make sure you know exactly what to build, and whom you’re building it for. It helps to ask a few questions up front to design the most effective dashboard possible, and avoid [dashboard rot](/content/blog/five-strategies-fight-dashboard-rot/index.html): - Why are you making this dashboard? - What does your audience want to see? - What do they know about the data, what do they expect? - What are the 3-4 key numbers or bits of data they need? - How can they dive deeper into the data? Building a dashboard is a balancing act between adding enough information and cluttering it up with too many charts and filters. Often, the easier solution is to just add more views, thinking that they’ll be useful to somebody – but they end up creating clutter for everybody. Asking questions helps make decisions about where to draw the line, which otherwise end up being arbitrary. Part of this step is also to figure out if you’re building one dashboard or if it needs to be split up into multiple dashboards or pages. It’s not uncommon to try to please many different stakeholders by adding many options to an existing dashboard, when breaking it up into multiple, simpler and more focused dashboards would be better. ## Designing dashboard layouts with a clear visual hierarchy > _Overview first, zoom and filter, then details-on-demand_ > > – Ben Shneiderman When building the dashboard, the first question is how to structure its overall layout. Ben Shneiderman’s “visual information-seeking mantra” (published in [a 1996 paper](https://www.cs.umd.edu/~ben/papers/Shneiderman1996eyes.pdf)) can serve as a high-level guideline here. Its intent was to guide people building data visualization applications, before they were known as dashboards. To guide the user through the different stages of overview and details, the dashboard needs to be structured to make it clear where they’re supposed to look first. In [this example dashboard showing electricity generation and consumption in the U.S.](https://observablehq.observablehq.cloud/framework-example-eia/), the map stands out. This is the first item users will look at. It provides an overview of the data, as well as the main control (the time slider). Maps are common anchors on dashboards, though they need to be used with care. Is the geographical aspect of the data really important, or is the map just a nice way to visualize data? Other anchors are big numbers (see below), or big charts. Color and shading can also be used to draw attention to the key items, as can larger font size. The trick is to decide what is more important, and what is secondary or tertiary. Trying to make everything big and colorful will only create chaos and overwhelm the viewer. A well-designed dashboard doesn’t yell at its users, but subtly helps them read and navigate the information through a clear visual hierarchy, and leads to faster insights and decision-making. ## Include big, important numbers when building dashboards When your viewers want numbers, give them numbers! We tend to want to show as much data as possible on a dashboard, so naturally we’re thinking of charts and [data visualizations](/content/learn/what-is-data-visualization/index.html). But simple numbers are important and can make a dashboard much more usable by making key values available at a glance. Big numbers also help with the visual hierarchy. They say, look here first! Your users probably know their key metrics, and have a solid frame of reference to know what numbers to expect (and which ones would be bad). Showing those at the top is a great way to give your users the most important information as quickly as possible. Take [our Plot example dashboard](https://observablehq.observablehq.cloud/framework-example-plot/), for example. It uses big numbers to highlight Observable Plot metrics from GitHub, like the latest release, stars, daily downloads, and total npm downloads. These numbers are of course reinforced by additional charts on the dashboard that reveal more detail and let viewers dig deeper. But without the highlighted numbers, users would have to spend more time figuring out if things are going well or not. ## Include coordinated views when building dashboards Dashboards are useful because they let viewers see information that needs to be considered together, all in one place. This can mean different aspects of the same (or related) data, say sales numbers and orders, or bookings and the weather. Another approach is to show overview and detail charts, where the overview indicates what part of the data is shown in the detail chart. This is illustrated very nicely in our [primary mortgage rates dashboard example](https://observablehq.observablehq.cloud/framework-example-mortgage-rates/). The bottom data visualization shows a long time period, from 1971 to 2024. The shaded area is shown in more detail in the top view. When the dashboard loads, it shows the last 12 months of data. The user can then drag the shaded area to see other 12-month periods in detail. But they can also resize the box to see a larger or smaller time period. Coordination is also important when filtering views, as we’ll discuss in the next section. When more views respond to interaction, especially when they show different data, users can get more information from the dashboard. As they drill into the data, users want to see as many of the views respond to show their selected subset of the data. Static views are still useful for overviews and as reference, but most views should respond to interactions such as [filtering](/content/blog/linked-brushing/index.html). ## Add filters for more interactive dashboards The most common interaction on dashboards are filters. Filters allow users to flip between different aspects of the data, or drill into subsets they’re interested in. Take [our hotel bookings dashboard](https://observablehq.observablehq.cloud/framework-example-hotel-bookings/), for example. It opens by showing all bookings. But at the very top, there is a filter control that lets the user pick subsets of the data: _direct_, _corporate_, _online travel agent_, _offline travel agent_, and _groups_. The overall overview is a good starting point, but most users want to see more fine-grained data. Filters let them pick the parts of the data they want to see. The key here is to find the right balance between giving users control and not overloading the dashboard with too many filters. Knowing your audience and their needs is important here. Simply adding any filter you can think of to a dashboard, or doing the same for every user request, might seem helpful, but usually leads to worse dashboards. When too many filters start weighing a dashboard down, it’s often better to break it up into multiple pages to serve the different users better. Another consideration here is the speed of interaction. If users have to wait too long for their actions to have an effect, they lose focus and won’t be able, or have the desire, to ask more questions. Making dashboards fast is therefore crucial. [Static site generators like Observable Framework have an advantage](/content/blog/what-is-static-site-generator/index.html) over traditional BI tools here, because they don’t require database access at load or when the user interacts. ## Coordinate colors for more intuitive dashboards Having multiple views on a dashboard is great, but it’s important to not overwhelm the audience or cause confusion. One common mistake that can lead to misunderstandings is the re-use of colors between different views. As a general rule, if a color is used in multiple data visualizations, it should mean the same thing. On [the hotel bookings dashboard](https://observablehq.observablehq.cloud/framework-example-hotel-bookings/), for example, the grass green color stands for summer, and the brown color represents fall. This is the same in the [donut chart](/content/blog/truth-about-pie-charts/index.html) at the top, and the bar charts across time further down. Color is a powerful visual cue, and it needs to be used with care. While the same color used for different things will lead to confusion, color used wisely to connect different charts will help viewers understand what is shown much more quickly. A common issue with colors is that we can only really differentiate a small number of them, around 10 different hues (we have [a blog post on how we designed our own set of colors](/content/blog/crafting-data-colors/index.html)). This limits the amount of different items that can be shown on a dashboard. The answer is not to reuse colors or use too many, but either to break up a dashboard into multiple pages, or use shades of colors within charts. ## Monitor and improve your dashboard As much as we might hope to be done once the dashboard is built, it’s really just the first version. A dashboard often needs some iteration with its users, after they’ve had a chance to use it, to get to its final state. It’s important to make sure your users know about the dashboard, and that you can tell if they’re actually using it. [Use analytics to track usage](/content/blog/how-to-track-dashboards-enhanced-analytics/index.html), and if you’re not seeing your intended users actually use your dashboard, find out why. This goes back to the initial point about asking questions, only now the answers will be more specific because your users have a point of reference. Once users actually get to use a dashboard themselves, they will see what they can do with it and what’s missing, have additional questions they want to ask, and perhaps realize that they need something quite different. How you deal with that information is of course up to you, but the goal is to provide value to your users, even if they don’t always know what exactly it is they’re looking for. ## **Conclusion: Helping users understand data through well-designed dashboards** Dashboards are powerful tools for showing and understanding data. Some simple considerations can go a long way to making dashboards and data visualizations more useful and effective for your users. Picking a good layout, using color correctly, adding interaction and coordinated views, as well as big numbers, all might seem like simple steps, but when done right, they end up being more than the sum of their parts. This is especially true if you’ve asked the right questions up front and continue to monitor and ask questions to make sure you’re building the right dashboard for your users. At the end of the day, the goal of a dashboard is to help your users understand the data and make the right decisions based on that data. Whether you're building dashboards for internal teams or external clients, following these principles will help you create dashboards that are not only visually appealing but also functional, intuitive, and effective. Our static site generator, Observable Framework, can help your team build and deploy data apps dashboards that load instantly — the first time, and every time. Learn more about building your first Framework data app with this step-by-step [tutorial](/content/framework/getting-started/index.html). \]\]>https://observablehq.com/blog/from-data-exploration-to-data-apps-with-observable https://observablehq.com/blog/from-data-exploration-to-data-apps-with-observableWed, 14 Aug 2024 00:00:00 GMTThere isn’t one right way to get value from data: the best tool for a data job is the one that helps you, your team, or your viewers to answer their questions — quickly, responsibly, and with confidence. Some questions require an ephemeral one-off SQL query or exploratory chart. For example, a VP of sales might ask about a single customer’s total page visits over the past 30 days. In that case, an analyst might perform a simple SQL query and spin up a basic line chart. Other questions, however, might require a persistent, polished dashboard intended for users to view frequently and over the long term. Most often, data teams need tools that allow them to do both: initial data exploration, prototyping, and quick tests, with key insights refined and shared in a final report or data app. Wherever your questions exist along that spectrum, Observable has you covered with [Notebooks](/content/platform/notebooks/index.html) and Observable [Framework](/content/framework/index.html). Notebooks are interactive, editable documents defined by code. They are created and hosted on the Observable platform. In notebooks, you can combine text in Markdown, code (in JavaScript, SQL, or HTML), and outputs (likeinteractive charts and tables) in one place. Observable Framework is a static site generator, with projects [composed of files](/content/framework/project-structure/index.html) and developed locally in your editor of choice to build fast, custom, user-facing data apps and dashboards. Observable users can work in browser-based Observable Notebooks (left), or build data apps locally with code using Observable Framework (right). Here, we describe where Observable Notebooks and Framework shine, and how – when used in combination – they accelerate insights from data along the entire spectrum of data work from initial exploration to production-ready data apps. ## Using Observable Notebooks for data exploration, collaboration, and _ad hoc_ analysis Observable Notebooks are designed for [data exploration](/content/learn/what-is-data-exploration/index.html), [data analysis](/content/learn/what-is-data-analysis/index.html), and [data visualization](/content/learn/what-is-data-visualization/index.html) prototyping. With Notebooks, you can easily upload data, add Markdown text and code (including JavaScript, SQL, HTML), create interactive data visualizations and other outputs, and share your explorations with your team. ### Exploring data and prototyping data visualizations with Observable Notebooks Observable Notebooks are ideal for data exploration, tinkering, and prototyping graphics. Notebooks live entirely in the browser, with common JavaScript libraries for data exploration and data visualization (like [Observable Plot](/content/plot/index.html) and [D3](https://d3js.org/)) available out-of-the-box as part of our [standard library](/content/documentation/misc/standard-library/index.html), so there’s no installation required to get started. Connecting to data in databases, local or cloud files, and APIs is fast and painless with existing database connectors and a friendly user interface to attach local or cloud files, or access data from APIs. With your data connected to a notebook, instantly see the contents and summary distributions with the [Data table cell](/content/documentation/cells/data-table?collection=%40observablehq%2Fgetting-data-in-and-out/index.html) or, for database tables, the [SQL cell](/content/documentation/cells/sql/index.html). A local CSV file is attached and inserted into a Notebook, where the contents and variable distributions are immediately visible in a Data table cell After taking a quick look at the data in table format, make exploratory charts in minutes with [Observable Plot](/content/plot/index.html) (our open-source library for exploratory data visualization). Choose from our selection of code snippets for common chart types, then substitute the placeholder data with your own. You can use other libraries in our standard library for data visualization, or choose from thousands of open-source JavaScript modules to use in notebooks with [require](/content/documentation/cells/require#require/index.html). A Plot snippet is inserted into a Notebook, then updated with the attached data to quickly create an interactive bar chart. Use notebooks to prototype charts that eventually become part of final data apps. Continue building out exploratory charts and adding interactivity with [Observable Inputs](/content/documentation/inputs/overview/index.html). Or, don’t start from scratch! Get inspired by thousands of stunning visuals that live in public notebooks created by Observable community members, fork to make your own copy, then drop in your own data and start fine-tuning. Explore thousands of public community examples in Notebooks, then fork to make a copy that you can adapt and fine tune for your own data and graphics. Just because notebooks are best suited for prototyping doesn't mean that it is impossible to do so in Framework. When building with Framework, the local development server lets you see live changes as you edit. And, Framework (like notebooks) supports [automatic reactivity](/content/framework/what-is-framework#automatic-reactivity/index.html), which means your code automatically re-runs whenever variables change so you're always seeing the latest version of all page content as you iterate. In fact, in some cases Framework can even be _better_ for prototyping charts. For example, if your chart depends on slow database queries, iterating on visuals can be bogged down by time-consuming client-side processing in notebooks. But with Framework’s [data loaders](/content/framework/loaders/index.html) (which we return to below), you can generate a minimal data snapshot that lets you iterate more quickly, and without pre-aggregation. Check out our [recent webinar](https://www.youtube.com/watch?v=n5gFBQTClxc&t=1457s) with Mike Bostock to learn how this helped us go beyond summary statistics to more deeply explore and develop rich visuals for our internal web logs. ### Collaborating on data visualization prototypes with Observable Notebooks Notebooks are the best place for teams to collaborate while exploring and developing data displays together. Features made for collaboration (like fine-grained [access control and sharing](/content/documentation/collaboration/sharing/index.html), [multiplayer editing](/content/documentation/collaboration/#multiplayer-editing/index.html), and [in-notebook comments](/content/documentation/collaboration/comments/index.html)) make notebooks a fast — even delightful — place for teammates and stakeholders to work together while keeping everyone’s code, eyes, and ideas literally on the same page. Quickly add collaborators, edit code together, and add comments in live browser-based Observable Notebooks. Of course, you can collaborate in Framework projects using more traditional channels and tools (e.g. sharing, reviewing, and merging work using development platforms like GitHub that support project management and collaboration). And, we’ve started live coding together when building in Framework using VS Code’s [Live Share extension](https://code.visualstudio.com/learn/collaboration/live-share). But when it comes to nimble collaboration during exploratory data analysis and visualization, we often look to notebooks first. ### Conducting _ad hoc_ analysis, computations, and queries with Observable Notebooks Some questions just need quick, _ad hoc_ analysis, computations, queries, or exploratory data visualizations. Despite their shorter lifespan and smaller viewership, those fast, on-the-fly data analyses are critical to incrementally adjust business understanding and expectations – and they are another place where notebooks shine. For example, when you securely connect a workspace to a database using our [database connectors,](/content/documentation/data/databases/overview/index.html) all members of the workspace with edit access can immediately start writing queries to answer questions and test hypotheses as they arise. With a database connected, quickly write queries in a SQL cell and see outputs immediately above in an interactive table. We often use notebooks for exploratory data analysis and _ad hoc_ queries internally, as described by Observable's Ananya Roy: > Having our internal databases connected to notebooks has meant that there is no barrier to answering questions with real data. This is very empowering. Whether we have a problem to investigate, a new metric to explore, or simply a curiosity to satisfy, it only takes a few seconds to open a notebook and start querying. Ananya Roy Customer Success Engineer at Observable Keep in mind that when working in notebooks, all your code runs in the browser – which means content can load slowly if it depends on time-consuming queries or data analyses. And, it can be challenging to fully customize display layouts and styling in notebooks. Read on to learn how Framework solves for data app performance and presentation. ## Building fast, custom data apps that fit into your developer workflow with Observable Framework Notebooks are great for exploration and prototyping, but as founder and CEO Mike Bostock [recently wrote](/content/blog/observable-2-0/index.html): ”a notebook can’t compete with a custom web app in terms of user experience.” We look to Observable Framework when we need to make performant, finalized, custom data apps and dashboards that provide viewers with clear insights and the best user experience. A selection of data apps and dashboards built with Observable Framework. ### Using Observable Framework for flexible developer workflows, speed, and polyglot support Observable Notebooks live on observablehq.com. Notebooks aren’t islands; you can [require JavaScript modules](/content/documentation/cells/require#requirements-for-require/index.html) and [import notebook content](/content/documentation/notebooks/imports/index.html), or [embed notebook content](/content/documentation/embeds/index.html) elsewhere. But a notebook is a web page, not a file, so can’t be easily integrated into file-based workflows. For example, notebooks can’t be tracked using git for version control, or bundled along with other files that are deployed together as pages in a website. Framework projects, on the other hand, are composed of local files (with defined [structure](/content/framework/project-structure/index.html) and clever routing), so they can slide right into your team’s existing file-based workflow. And, since it’s _just files_, you can build in whatever code editor you choose. Work in any code editor (like Visual Studio Code, above left) while building data apps in Framework, and see page updates as your work locally in live preview (above right). Now, let’s talk languages. It’s difficult to integrate languages that teams most often use for [data wrangling](/content/learn/what-is-data-wrangling/index.html) and analysis, like Python and R, with web-based Observable notebooks. Despite recent WebAssembly (WASM) developments that make it possible to execute code in other languages directly in the browser (e.g. [WebR](https://docs.r-wasm.org/webr/latest/) and [pyodide](https://pyodide.org/en/stable/)), it’s far from simple _._ As [Jeffrey Perkel recently wrote](https://media.nature.com/original/magazine-assets/d41586-024-00725-1/d41586-024-00725-1.pdf) regarding WASM for scientific computing: “brace yourself for complications.” Framework’s solution to the polyglot problem is [data loaders](/content/framework/loaders/index.html). Data loaders are programs that generate static data snapshots at build time. They can be written in _any_ language — Py, R, Rust, Go, Ruby, SQL, or whatever else — to access, wrangle, and analyze data. Because heavy data processing and minimization is done at build time (instead of when a user opens a page), that means faster data apps and happier viewers. Learn more about the benefits of data loaders in our [recent blog post](/content/blog/data-loaders-for-the-win/index.html) and explore our reusable [data loader examples](https://github.com/observablehq/framework/tree/main/examples#data-loaders). In summary: local, file-based Framework projects can more easily integrate with your existing data and developer workflows, let you work in your preferred editor, and support data loaders written in any programming language. If you’re looking for a flexible way to build that plays nicely with your current toolkit to produce fast and custom apps, Framework is a good way to go. ### Build the best custom, production-ready data apps and expressive visualizations with code When you build with Framework, you can easily arrange charts, tables, and text in denser displays typical of dashboards using the built-in [grid and card classes](/content/framework/markdown#grids/index.html). Quickly update the overall look of one or all pages using our [dark and light mode themes](/content/framework/themes/index.html), which will automatically respond to your viewers’ device settings. Or, create your own [custom stylesheet using CSS](/content/framework/config#style/index.html) to bring your organization’s branding to a project. A selection of Framework's built-in dark and light mode themes. Use Framework’s [configuration options](/content/framework/config/index.html) to add custom headers and footers, control the table of contents, show or hide the sidebar by default, and more. Like in notebooks, you can create bespoke, interactive visuals in Framework using libraries like [D3](/content/framework/lib/d3/index.html), [Observable Plot](/content/framework/lib/plot/index.html), [Mosaic vgplot](/content/framework/lib/mosaic/index.html), [deckgl](/content/framework/lib/deckgl/index.html), and others that exist out-of-the-box as part of our standard library. Or, [import](/content/framework/imports/index.html) any other JavaScript libraries from npm, node, or import local or remote modules to use your preferred tools. So if you need to create custom, polished data apps that provide viewers with a better final presentation and user experience than what’s easily possible with notebooks, we recommend building with Framework. ## The best option? Use both: Notebooks for collaboration, and Framework for building finished data apps Many data projects can benefit from working in both Observable Notebooks and Framework. We often do our exploratory data exploration, visualization, analysis, and prototyping together in notebooks, where our team can build and prototype more nimbly. Once we’ve built out our first chart drafts, we transfer the code into a Framework project for fine-tuning and iteration, custom layouts, performance optimization, and deployment as a final data app. Currently, we either transfer notebook prototypes to Framework files with a simple copy and paste, or, for public notebooks, using Framework’s built-in [convert command](https://observablehq.observablehq.cloud/framework/convert) (beta). Within an existing Framework project, you can use the convert command as shown below to convert the [Scatterplot notebook](/content/@observablehq/plot-scatterplot/2/index.html) from the Plot gallery: ``` npm run observable convert 'https://observablehq.com/@observablehq/plot-scatterplot/2' ``` Or, if you're using yarn: ``` yarn observable convert 'https://observablehq.com/@observablehq/plot-scatterplot/2' ``` The convert command helps you convert content from one or more notebooks to standard Markdown for use in Framework projects. While this doesn’t completely eliminate the need for manual tweaks (for example to convert from Observable JavaScript used in notebooks to Framework-supported vanilla JavaScript, or to port over imports from Observable Notebooks, which is currently not supported), it does get you most of the way. Visit the [convert command documentation](https://observablehq.observablehq.cloud/framework/convert) to learn about other caveats, options, and tips for adapting your notebook code for Framework data apps. We’re excited to improve how users can work between Observable Notebooks and Framework. For example, we’ll eventually port vanilla JavaScript into notebooks, and will extend the convert command to work with private notebooks. ## Get started with building data visualizations on Observable Whether you’re working in Observable Notebooks for collaborative data exploration and exploratory data analysis, in Framework to build faster custom data apps, or both, we have resources to help you get started: - [Notebooks and platform features](/content/documentation/index.html) documentation - [Framework documentation](/content/framework/index.html) and [Getting started tutorial](/content/framework/getting-started/index.html) - Join us on our Observable [Community Slack](https://observable-community.slack.com/join/shared_invite/zt-2e65h19s0-sDGOzLrShYHys9ibzBakuw#/shared-invite/email) and [Forum](https://talk.observablehq.com/) to ask questions, share progress, and stay up-to-date on events, product releases and new resources \]\]>https://observablehq.com/blog/five-underused-charts-for-richer-dashboards https://observablehq.com/blog/five-underused-charts-for-richer-dashboardsThu, 01 Aug 2024 00:00:00 GMTImagine a typical dashboard, built either for customers or for tracking your organization’s internal metrics. What types of charts come to mind? Chances are you’ve envisioned some usual suspects — bar charts, donut or pie charts, line charts, a basic map, and big number boxes. While these are extremely common and useful chart types, in some cases their simplicity can leave data insights, and user engagement, on the table. Here, we describe five chart types that squeeze more juice from your data to create richer and more engaging dashboards. ## **Horizon charts save space and highlight trends** A [horizon chart](https://en.wikipedia.org/wiki/Horizon_chart) provides a compact way to show patterns for a quantitative variable, usually over time. You might be thinking “Wait, can’t I do that with a simple line or area chart?” Technically, yes. But sometimes, you need an option that still clearly shows trends over time, but minimizes the space needed to show them. In a horizon chart, the range of values is split into equal bands. The bands are layered atop one another, using color or opacity to distinguish between them, as shown in the animation below. This band layering makes the most of limited vertical space, while clarifying trends and adding visual interest. Usually, horizon charts are faceted to compactly show trends across multiple groups. For example, the chart below visualizes trends in unemployment by industry using data from the U.S. Bureau of Labor Statistics: See the Observable Plot code used to create this [horizon chart](/content/@observablehq/plot-unemployment-horizon-chart/index.html). In the chart above, color is used to differentiate between the layered bands, with darker greens and blues punctuating periods of higher unemployment within an industry. This highlights patterns in unemployment over time. For example, we can clearly see seasonal cycles in unemployment for Construction and Government industries, but these cycles are offset by about six months. And, we can see similar unemployment patterns over time for Wholesale and Retail Trade, Manufacturing, Leisure and Hospitality, and Business Services, each having a moderate increase from about 2002 to 2006, followed by a decline that leads into a sharp uptick in unemployment near the start of 2009. For comparison, a standard line chart showing the same unemployment data is shown below. The number of overlapping lines makes it a bit overwhelming to track patterns for any one industry, or to compare seasonal or longer term patterns between industries with unemployment numbers on very different scales. Given the spaghetti-esque nature of the line chart above, one might consider faceting by industry, as shown below. But this faceted version of the chart — when sized reasonably for viewers to fit in the same window — compresses unemployment fluctuations onto a short vertical scale for each industry, making it difficult to interpret all but the most obvious changes. The horizon chart we started with solves both problems seen in line chart alternatives: you can still use faceting to clarify trends by group, and the layered bands accentuate trends, even when presented in compressed vertical space. One thing to keep in mind when building a horizon chart is that viewers are unlikely to have a deep understanding of band layering, so it may not be the best option if you want them to correctly interpret the _exact_ _values_ shown in the chart. But, if the aim is to help your users see overall patterns _and_ if you have multiple groups and limited space, a horizon chart can be a great option. ## **Beeswarm charts** Beeswarm charts are used to visualize the distribution of observations along a single axis, using a [dodge transform](/content/plot/transforms/dodge/index.html) to closely pack the points while avoiding any overlap between them. Beeswarm charts are useful when you want to represent unique points, while making it easy for viewers to distinguish between them. For example, the beeswarm chart of FDIC-reported bank failures below — which also maps bank size to dot radius — clearly captures the massive number of bank failures (across bank sizes) starting in 2008, and the much different spike from several very large bank failures in 2022. Conditional text labels make it easy to identify major bank failures, while interactive tooltips let a user dig into the smaller bank failures without cluttering the entire chart with text. Explore the [interactive version and Plot code to create this chart](/content/@mbostock/bank-failures/index.html). You can represent additional quantitative or qualitative values for each point in a beeswarm chart by mapping them to things like color, size, or opacity. For example, the Washington Post recently compared [ethnicities of federal judges appointed by Trump and Biden](https://www.washingtonpost.com/politics/2024/05/17/biden-trump-judges-diversity/), using fill color to represent ethnicity, image backgrounds to highlight Supreme Court justices, text annotation to highlight appointments of interest, and with court type mapped to dot size: Beeswarm chart representing ethnicities of federal judges appointed by President Joe Biden. Source: Nick Mourtoupalas. “ [Biden has installed the most non-White judges of any president](https://www.washingtonpost.com/politics/2024/05/17/biden-trump-judges-diversity/).” The Washington Post. May 22, 2024. As the chart above illustrates, mapping variable values to visual properties makes it possible to create clear, engaging beeswarm charts that are rich with information, helping viewers to learn more about distributions than might be possible from [histograms](https://en.wikipedia.org/wiki/Histogram) or [density plots](https://en.wikipedia.org/wiki/Density_estimation). But beware: beeswarm charts can fail if you have too many points, which (in order to keep dots from overlapping) can send the distribution skyrocketing beyond your chart area. And, keep in mind that the dodge transform to avoid overlap means that beeswarm charts should not be used to infer exact values along the axis. ## **Combine maps, marks, and interactivity** As open-source, code-based data visualization libraries like Observable Plot increasingly add support for spatial data visualization, you can design custom maps to go beyond what’s possible with many point-and-click GIS tools. You’re no longer confined to bubble maps and univariate choropleths (which are still great options). If you’re looking to mix things up, here are a few map variants to consider. For an alternative to a [bubble map](/content/@observablehq/plot-us-bubble-map?intent=fork/index.html) to show variable values at different locations, consider a spike map. The spike map below is created by combining the [geo](/content/plot/marks/geo/index.html) and [vector marks with a spike shape](/content/plot/marks/vector#spike/index.html) to visualize 2016 U.S. population by county, using data from the [American Community Survey](https://api.census.gov/data/2016/acs/acs5/cprofile/examples.html). See the code to create this [spike map](/content/@observablehq/plot-spike?intent=fork/index.html) with Plot. A spike map can be preferable to a bubble map if you want to avoid largely overlapping dots. And, spike length can be easier for viewers to interpret and compare across locations, compared to dot radius. What if you want to represent trends or changes in spatial data? Three options are to add interactivity, show map snapshots as small multiples, and use an arrow mark to represent a direction and magnitude of change. For example, the interactive choropleth below reveals the increase in broadband access by county across the U.S. from 2000 to 2018. Follow along with [our tutorial](/content/@observablehq/build-your-first-choropleth-map-with-observable-plot/index.html) to create this interactive chart on your own using Observable Plot and [Observable Inputs](/content/documentation/inputs/overview/index.html). For a static alternative, consider showing maps as small multiples (here, showing [new Walmart stores added by decade](/content/@observablehq/plot-map-small-multiples?intent=fork/index.html)): Or, use an [arrow mark](/content/plot/marks/arrow/index.html) to indicate a magnitude and direction of change by location. The chart below uses an arrow mark to visualize the margin by which each presidential candidate (Joe Biden = blue, Donald Trump = red) won for each U.S. county in the 2020 election. See the code to create this [election wind map](/content/@observablehq/plot-election-wind-map/index.html) with Plot. As a final example of building better custom maps with code, let’s consider a case where you want to aggregate or simplify how you visualize spatial data. The two maps below summarize spatial data on Walmart store locations, but in different ways. In the left chart, a [density mark](/content/plot/marks/density/index.html) highlights regions of high (dark blue) and low (light green) Walmart store densities. On the right, a [hexbin transform](/content/plot/transforms/hexbin/index.html) also summarizes store densities, with larger hexagons indicating a greater number of stores near that location and with color indicating the date that the first nearby store opened. Many combinations of maps, marks, and interactivity are possible beyond what we’ve shown here. This selection gives a taste of what you can make by merging different chart marks with spatial data. ## **Mosaic plots put proportions in another dimension** If a stacked bar chart to show proportions by category has left you feeling one dimension short, a mosaic plot might help. [Mosaic plots](https://en.wikipedia.org/wiki/Mosaic_plot) (also called Marimekko or Mekko charts) visualize proportions along two dimensions. They are often used to visually represent values in a [contingency table](https://en.wikipedia.org/wiki/Contingency_table). In the mosaic plot below, proportions of sales by segment (Delicious-n-new, Carrots-n-more, etc.) are shown on the y-axis, with markets along the x-axis. This looks very similar to a standard stacked bar chart (with city on the x-axis), with one major difference: instead of each bar having the same width, the bar width is variable based on market size. See the code to create this [mosaic chart](/content/@observablehq/plot-marimekko/index.html) with Plot. The variable width along both axes allows a viewer to consider parts of a whole based on two variables. We can see, for example, that _Auburn, AL_ is the largest of the four markets (occupying the greatest width). Within that market, _Almond lovers_ accounts for the most sales (the highest proportion on the y-axis). The area of the _Auburn, AL_ _/_ _Almond lovers_ rectangle in the bottom left is larger than all other rectangles in the chart, indicating that it accounts for the greatest sales overall. With more segments and smaller rectangles, it may not be feasible to label each part of a mosaic plot. In that case, you may want to limit labels to only highlight major or notable values. For example, in the faceted mosaic chart below (using the same market data as above, but split over three decades), only the largest rectangles are directly labeled.  Interactive tooltips let a viewer explore values for smaller rectangles, while avoiding clutter and overlapping text. A mosaic plot can also be useful to: - Compare budget allocated by **city** and **department** - Visualize electricity usage by **building** and **source** (e.g. solar and natural gas) - Explore **support for a bill** (yes / no) based on **age brackets**(e.g. 18 to 30 years, 31 to 50 years, 51 to 70 years, etc.) using data from a poll of likely voters for a region But use mosaic charts with care — they can be difficult for viewers to interpret because the different aspect ratios of rectangles make it hard to compare areas, especially when the areas are similar. You can help viewers by adding text annotation to highlight the main takeaways, and including the actual values for each area either on the chart or with interactive tooltips that reveal values on hover. ## **Track how you’re keeping up with burndown charts** Often, teams want to track user requests and understand how efficiently they are resolved. A burndown chart shows the number of outstanding issues over time. The issue creation date can be mapped onto a color scale, revealing how quickly issues are resolved. The example below by [Tom MacWright](/content/@tmcw/index.html) is a [burndown chart of GitHub issues](/content/@tmcw/github-burndown/index.html) filed in the Observable Plot repo: In the burndown chart above, the total height of the chart area indicates the number of open issues at any time. Color is used to represent the date an issue was opened. The color channel is useful because it allows a viewer to track the “lifetime” of open issues based on the general time window when they were filed. For example, a spike in issues from about April 2023 to October 2023 is captured by an “orange cohort”. The relatively constant scale of that band indicates that most of these issues remain open. Compare that to the “blue cohort” of issues created between March and July 2022, which have steadily reduced over time. Open repo issues are a common burndown chart use case for software developers, but this type of chart can also be used to track things like outstanding customer support tickets, incomplete project tasks, or unanswered emails. If you want to visualize both how many to-dos are still open at any point _and_ how quickly things get resolved, a burndown chart is a useful option. ## **Start building richer dashboards** Here, we’ve highlighted lesser-used charts that can help dashboard users get deeper insights from data than they might from more commonly used graphics. When you build charts and dashboards with code, for example using [Observable Plot](/content/plot/index.html) (our open-source library for exploratory data visualization used in many of the examples above) and [Observable Framework](/content/framework/index.html) (our open-source static site generator for final data apps), you can fully customize your data displays for more useful and engaging dashboards that keep users coming back. Start building richer dashboards today: - Visit the [Plot gallery](/content/@observablehq/plot-gallery/index.html) for more chart ideas, inspiration, and reusable examples - Learn more about [Observable Framework](/content/framework/index.html) for building fast, custom data apps and dashboards \]\]>https://observablehq.com/blog/observable-makes-building-data-apps-faster-and-easier https://observablehq.com/blog/observable-makes-building-data-apps-faster-and-easierTue, 23 Jul 2024 00:00:00 GMTBuilding custom data apps and dashboards typically involves many steps: meeting with various stakeholders to get their requirements, finding the right data sources, and building prototypes just to get started. Then iterating on the custom data app, going through change requests, and finally maintaining the resulting data app or dashboard for years to come. Observable Framework helps with many of these steps, with tailored solutions for developing and maintaining a dashboard or data app. ## **Build modern data apps and dashboards that your customers will use** Users today expect their dashboards to look and work like the apps they’re used to on their phones: clean and modern looking, seamlessly working across different device and screen sizes, and – most of all – fast. A dashboard that is slow, clunky, or is impossible to use on a phone will frustrate your users. Fast load times aren’t just a nice-to-have, but a real competitive advantage. If your dashboard is facing external users or customers, its performance and visual appearance will color their perception of your product. When today’s established [business intelligence](/content/learn/what-is-business-intelligence/index.html) (BI) tools were new, direct access to databases was novel and exciting. But in the decades since, the wait for fresh data has turned into a tedium that users aren’t used to anymore from other data-focused apps. A different approach is needed today, one that presents data immediately, no matter how slow the backend database it’s reading from might be. Data apps built with Observable Framework are much faster than traditional dashboards because of data loaders. By moving database access, data transformations, modeling, etc., to the build phase, the experience for the user is an app that loads instantly. We have [a blog post that explains how data loaders work (and why they’re awesome)](/content/blog/data-loaders-for-the-win/index.html), but the gist is that they load and transform the data offline, long before the user requests the page. Their output gets bundled with the code and sent to the browser very efficiently, without having to wait for a database query or further processing. Interaction in a Framework app can also be much faster by pre-loading the data needed for drill-down, different filter settings, etc. All of these operations are then performed right there in the browser. Traditional dashboards respond to interactions by sending a new query to the database to get a result set based on the new settings. Framework apps can perform the same operations in memory using [DuckDB](https://duckdb.org/) or similar databases. Efficient data formats like [parquet](https://parquet.apache.org/) that can easily transmit millions of rows to the user, and allow the app to query them much faster than the round-trip time of a query on the network. ## **Streamline data app and dashboard development with modern web standards and advanced features** Application speed aside, an important consideration when choosing a framework for development, is how much it helps developers build their applications faster and more efficiently. Or in other words, how much time — and thus, money — they save by using it. The Observable approach, of using data loaders to push slow data queries to an offline process, enables developers to build data apps that are always up to date, load instantly, and work on any device their users might choose to use. ### **Accelerate data app development with reactive JavaScript** Framework’s reactive style of JavaScript encourages a modular approach to developing data apps and dashboards, which helps with both coding and testing. Any changes to values propagate through your code without having to write explicit update logic. Data updates, filters, and all kinds of interaction are much more straightforward to build this way. Developers familiar with React and other reactive frameworks will feel right at home, and can even use React code ( [JSX-style](/content/framework/jsx/index.html)) in Framework directly. ### **Utilize responsive layout and built-in components for faster development cycles** Observable Framework comes with a set of pre-built components and tools that help you get started quickly. These include layout components for responsive layouts, and interactive elements to add filters and other functionality to your dashboard. A modern data app needs to work across many different devices and screen sizes, from large monitors to smartphones. Resizing a dashboard is not just a matter of scaling individual parts of it, but also reflowing the layout to work on different aspect ratios and hiding less important elements on [smaller screens](/content/blog/big-insights-small-spaces/index.html). Modern browsers support the relevant cascading style sheets (CSS) features to make all this happen seamlessly, but configuring them is not straightforward. Framework’s layout helpers make this task much easier and faster. Framework’s set of recommended libraries, which are available without an explicit import, include many useful open-source ( [D3](https://d3js.org/), [Observable Plot](/content/plot/index.html), lodash) and custom libraries (like [hypertext literal](/content/framework/lib/htl/index.html)). For data loaders, we also have [a set of data integrations](/content/data-integrations/index.html) for you to quickly access your data from many sources. These have been honed over years of developing web applications, and provides a great starting point for your own work without having to go hunt for the right npm module for many common tasks. ### **Streamline deployments and easily control access** On the backend, you don’t want to waste time finding a way to control access (and you certainly don’t want to roll your own). [Observable Framework apps can be deployed](/content/framework/deploying/index.html) using Github Actions, to be hosted on Github Pages or another hosting provider. Framework app builds are easy to automate so they update and deploy as needed (we have [a blog post on the considerations for how often to build](/content/blog/align-dashboards-with-scales-of-business-decisions/index.html), based on your operation’s time scale). ## **Observable data apps are easier to maintain** Once you’ve built your data app, it is important to maintain it to keep it from breaking, and from falling into disuse — [what is commonly called dashboard rot](/content/blog/five-strategies-fight-dashboard-rot/index.html). Crucially, your data needs to stay up to date and your data sources remain accessible. How do you know if a database has gone missing, or permissions changed to cut off your users? Framework’s build process will throw errors when it can’t update its data, which you can then respond to immediately. Most BI tools will just silently fail, with your users being left in the dark about why they can’t see their data — and you as the developer never even finding out. Developing and maintaining a data app is not just a one-person job. Framework projects can easily be developed together (using common source-control tools like git and GitHub), and handed over to others who can take over monitoring deployments or maintenance when things go awry. Even if the original developer might have moved on or is working on other projects, the source code and project analytics are available to others who can maintain or extend the data app. This streamlined data app development process avoids having to rebuild from scratch or redo work that has already been invested. ## **Conclusion** There was a time when it was acceptable for business analytics software to be clunky, slow, and ugly. But today’s users expect modern, fast, and clean data apps that work wherever they are. Observable Framework makes modern web technologies available to developers to use in their data apps and dashboards. It saves time and money by making it easier to build and maintain data apps, and ties in with hosting and access control on the platform of your choice. And it delights users with apps that are fast and pleasant to use. \]\]>https://observablehq.com/blog/takeaways-from-building-showcase-dashboards https://observablehq.com/blog/takeaways-from-building-showcase-dashboardsThu, 11 Jul 2024 00:00:00 GMTThe Observable team recently developed [six example dashboards](https://github.com/observablehq/framework/tree/main/examples#showcase-examples-%EF%B8%8F) with the aim of highlighting what’s possible when you build with code in [Observable Framework](/content/framework/index.html), our open-source static site generator for data apps. Compared to traditional development workflows, Observable Framework offered: - Flexibility, which made building with Framework feel familiar — even for new users - Simplified project organization and reusable components - Unlimited customization through building data apps and visualizations with code By building with Framework, developers can more easily develop fast data apps, custom dashboards, interactive [data visualizations](/content/learn/what-is-data-visualization/index.html). Here’s what our team found when building data apps with Observable Framework: ## **Takeaway \#1: Flexibility makes a new tool feel familiar** There is always a learning curve when adopting a new tool. The shape of that curve depends, in part, on how quickly a user can map their existing knowledge, skills, and practices onto that new tool. When developing our first showcase examples, we (the Observable team) were early Framework users and testers ourselves — it was brand new to us, too! Framework’s flexibility helped us get started quickly, because we didn’t have to adopt an _entirely_ new suite of tools to start working with this one. Here are a few ways we didn’t have to start from square one. ### **Working with Observable Framework offers a better developer experience** One reason that Framework felt familiar is because we could each work in our preferred IDE or code editor, so we didn’t have to orient ourselves to a new platform or environment. Most of us worked in [Visual Studio Code](https://code.visualstudio.com/), some in [Zed](https://zed.dev/), and one teammate even tested in [Vim](https://www.vim.org/). Since each project is just code within a bundle of files, you can work in any editor you choose — which makes it faster and lowers friction to get started. And, because Framework projects are _just files_, we could use our existing workflows for versioning, collaboration, and project management (in our case, with git and GitHub). ### **Observable Framework offers polyglot options for data analysis and visualization** [Data loaders](/content/framework/loaders/index.html) let us access, wrangle, and analyze data in any language to prepare data snapshots at build time. The option to work in different languages helped us carry our existing skills (some of us in R, some in JavaScript, some in TypeScript, etc.) into Framework to prep data faster. This polyglot flexibility is captured in data loaders throughout our showcase examples: - In the [U.S. dams showcase dashboard](https://observablehq.observablehq.cloud/framework-example-us-dams/), we use an [R data loader](https://github.com/observablehq/framework/blob/main/examples/us-dams/src/data/dam-simple.csv.R) to access and wrangle records for 91k+ dams in the U.S. - The [Plot showcase example](https://observablehq.observablehq.cloud/framework-example-plot/) uses [TypeScript data loaders](https://github.com/observablehq/framework/tree/main/examples/plot/src/data) to pull metrics (like downloads, issues, etc.) for a GitHub repo - We wrote [JavaScript data loaders](https://github.com/observablehq/framework/tree/main/examples/eia/src/data) to access electricity demand and interchange data for the [EIA dashboard](https://observablehq.observablehq.cloud/framework-example-eia/) To see data loaders in other languages, check out our collection of [technique examples](https://github.com/observablehq/framework/tree/main/examples#data-loaders). ### **Choose your own data visualization libraries** Most charts in our showcase examples are built with [Observable Plot](/content/plot/index.html). For example, the interactive map featured in the [EIA dashboard](https://observablehq.observablehq.cloud/framework-example-eia/) below is built with Plot, while the range slider is created using the [range input](/content/framework/inputs/range/index.html) from Observable Inputs: For some charts, however, we needed a different tool. To create the donut charts in our Hotel bookings example, we used [D3](https://d3js.org/): And, to make a zoomable and scrollable map of U.S. dam locations, we decided on [deck.gl](http://deck.gl/): The three data visualization libraries mentioned above (Plot, D3, and deck.gl) are part of the Observable standard library, which means they are available out-of-the-box in Framework. But you’re not restricted to what’s in the standard library: you can [import and use any JavaScript library](/content/framework/imports/index.html) in a project, giving you freedom to build with whatever data visualization tools you prefer. ### **Computational notebook experience makes building Framework data apps feel familiar** We each had previous experience working in browser- and files-based computational notebooks (most extensively in [Observable notebooks](/content/documentation/notebooks/index.html), but also in Jupyter Notebooks, R Markdown, and Quarto, among others). That made building pages that combine Markdown, code blocks, and rendered outputs feel like familiar territory. For example, below is the first bit of raw code used to create our [Hotel bookings](https://observablehq.observablehq.cloud/framework-example-hotel-bookings/) showcase dashboard. As in other computational notebooks, a bit of YAML atop the .md file (bounded by three dashes) sets a number of page options (here, removing a default table of contents and setting the light and dark mode themes). Text added as Markdown is interweaved with JavaScript code blocks to import components and data. See the entire [index.md file used to create the page](https://github.com/observablehq/framework/blob/main/examples/hotel-bookings/src/index.md?plain=1). ## **Takeaway \#2: Using components simplifies project organization** When we first started building our showcase dashboards, we wrote a lot of our code directly in JavaScript code blocks within .md files. That changed as we saw our page source code get long, complex, and sometimes repetitive, which complicated editing and collaboration. We embraced components — reusable, smaller bits of code that are created as standalone scripts then [imported](/content/framework/imports#local-imports/index.html) into pages — to modularize an declutter page content. Our showcase examples show how we’ve moved toward componentized projects: - In our [Hotel bookings dashboard](https://observablehq.observablehq.cloud/framework-example-hotel-bookings/), we import separate JS components used to create [big number boxes (in HTML)](https://github.com/observablehq/framework/blob/main/examples/hotel-bookings/src/components/bigNumber.js) and [donuts charts (made with D3)](https://github.com/observablehq/framework/blob/main/examples/hotel-bookings/src/components/donutChart.js) - For the [EIA dashboard](https://observablehq.observablehq.cloud/framework-example-eia/), we export charts created as functions in the [charts.js](https://github.com/observablehq/framework/tree/main/examples/eia/src/components) file - The scatterplot and histogram in our [API example](https://observablehq.observablehq.cloud/framework-example-api/) are created as [individual components](https://github.com/observablehq/framework/tree/main/examples/api/src/components) We don’t aim to eliminate (or even minimize) all JavaScript code within a markdown file. However, moving big chunks of code (e.g. for charts or tables) into smaller components makes for [easier editing, reuse, and content layout](/content/blog/three-ways-build-better-dashboards-data-apps-with-code#reusable-components-to-save-time-and-effort/index.html). And, because Framework automatically watches local modules during live preview, any changes you make to components are reflected in the preview server. The schematic below shows one way a user might componentize a project: While we’ve found components useful, they’re not mandatory. You can write all page content right within your .md, like we have in our [Mortgage rates](https://github.com/observablehq/framework/blob/main/examples/mortgage-rates/src/index.md?plain=1) dashboard. Or, you can decide to write some charts as components (e.g. if you know they’ll be reused in multiple places) while others are created right in a page, as we’ve done in the [Hotel bookings](https://github.com/observablehq/framework/tree/main/examples/hotel-bookings) example. ## **Takeaway \#3: Building data visualizations with code offers unlimited customization** Framework’s built-in classes (e.g. [grid](/content/framework/markdown#grids/index.html) and [card](/content/framework/markdown#card/index.html)), helper functions (e.g. [resize](/content/framework/javascript#resize(render/index.html))), and [themes](/content/framework/themes/index.html) helped us quickly create nice first versions of each dashboard. But we wanted to take customization further for our showcase examples. Because everything in Framework is built with code, we could fine-tune without constraints. The examples below highlight three ways we’ve used code to fully customize our showcase examples. ### **Combine data in novel ways to unlock insights** For a number of our showcase dashboards, we wanted unique components that combine charts, text, and/or tables in novel ways. For example, the [Mortgage rates](https://observablehq.observablehq.cloud/framework-example-mortgage-rates/) dashboard features the custom big number box below, which combines HTML headers and a customized table with a tick chart to concisely show mortgage rate metrics and trends in one place: (Explore the [source code](https://github.com/observablehq/framework/blob/main/examples/mortgage-rates/src/index.md?plain=1#L21) for the component above.) ### **Customize chart legends with code** Sometimes a default legend just doesn’t cut it. That was the case for our EIA showcase example, which requires a legend combining a continuous color scale, custom value formatting (percentages), and discrete categories: Instead of being locked into an out-of-the-box legend, we were able to code up the one we wanted from scratch in JavaScript using Plot (see the [full code for the map and legend](https://github.com/observablehq/framework/blob/main/examples/eia/src/components/map.js)): ### **Framework offers interactivity and animation** Each of our showcase examples features at least one type of interactivity, like radio buttons, searchable tables, value sliders, and custom tooltips. In the [Mortgage rates](https://observablehq.observablehq.cloud/framework-example-mortgage-rates/) dashboard, we combine Plot and D3 to add brushing so that a user can zoom and drag over the line chart to focus on mortgage rates over a specific time period (see [the code](https://github.com/observablehq/framework/blob/main/examples/mortgage-rates/src/index.md?plain=1#L129-L172)). ## **See what else is possible, and start building custom data apps with code** Developing our showcase examples was a great opportunity for our team to test, learn, and implement strategies for creating custom data apps and dashboards with Framework. Our showcase dashboards and the takeaways above highlight _what_ and _how_ you can build with Framework, capturing the flexibility (with tools, workflow, and organization) and customization possible when you build entirely with code. Explore the [live showcase dashboards](https://github.com/observablehq/framework/tree/main/examples#showcase-examples-%EF%B8%8F) and source code to learn more. Then, follow along with our resources to start building your own: - [Getting started tutorial](/content/framework/getting-started/index.html) - [Observable Framework documentation](/content/framework/index.html) \]\]>https://observablehq.com/blog/what-is-static-site-generator https://observablehq.com/blog/what-is-static-site-generatorTue, 02 Jul 2024 00:00:00 GMTYou may have heard that [Observable Framework](/content/framework/index.html) is a static site generator. But what does that mean and why would that be a good thing? In short, it makes the experience for the users of your dashboards and data apps better by making them load much faster. It also reduces load on shared resources like databases, by querying data only once that might be used by many different people. In addition, it gives you more control over when the data is refreshed, which can be quite critical and is often overlooked. _This article is also_ [_available as a video here_](https://youtu.be/FQL2J7809a8) _._ ## How a traditional business intelligence tool loads a dashboard Let’s look at how a dashboard loads in your browser when you use a traditional business intelligence (BI) tool. To get the dashboard, your browser has to first request a page from the server. The server, in turn, has to figure out what goes onto that page. There are at least two parts to this, the structure of your dashboard and the actual data. The structure can live in a database or a file, so the web server has to load that information now. If you’re familiar with content management systems like WordPress, they work the same way – and have the same performance issues. Once it knows the structure, it can now query the big database for the actual data to be shown. This can also take a while because production databases aren’t usually optimized for analytical queries. It then has to run any data processing on it to create the result set that the browser needs to render the dashboard. Finally, the entire thing gets packaged up and returned to the browser, which can now render the dashboard for the user. But the user kicked off this whole process and then had to wait for it to complete. ## The "static" part of a static site generator A static site generator like Observable Framework works very differently. A build process runs independently of the user. It accesses the code and data, which might be coming from files and databases. It can run lengthy queries, complex data transformations, etc., that may take a long time to complete. The results of this process then get packaged up into a small package that contains the code and result data. This is the “static” part of a static site generator, because Framework generates a file, or a small handful of files, that contain the product of all of these processes in the same shape as what would be sent to the user’s browser. It’s also usually quite a bit smaller because it can be aggregated to the appropriate level of detail, and optimized for space efficiency. Now when the user requests the dashboard or data app, all the server has to do is access that package and send it back to the user’s browser, which then uses it to render it onto the screen. This is much faster because all the slow operations have already happened without the user having to wait for them. Now just because this is a static site, doesn’t mean that it can’t still access databases from the browser directly. In fact, because Observable Framework uses code, you have all the flexibility to do whatever you want here. This also includes querying data the dashboard was sent from the server in the package, to drill down, change filters, etc. But the key advantage is that even if many people request the same dashboard, that doesn’t generate more load on shared resources like databases. They all get served the same static package, and they don’t get slowed down by other people querying the exact same data from a database. ## Scheduled builds for controlled data updates Now you might wonder how this works if you want your data to be up to date. The key here is that for most use cases, changes to your data during the day or even during a week or month aren’t material and can be more of a distraction than actually help make decisions. We have a blog post about this, [_Better dashboards align with the scales of business decisions_](/content/blog/align-dashboards-with-scales-of-business-decisions/index.html). But a common problem is the appearance of a trend when there’s just incomplete data. Like in this case above, where we might have an incomplete day or week at the end. Once we’re past this point and the data is complete, it might look quite different. The answer to this is scheduled builds that run at certain intervals. These can be short, for example you might run your builds every hour. But more likely, you’re going to run them once a day, week, month, or even quarter. This is certainly not the case for all data, but for the vast majority of business use cases, a properly managed process of scheduled updates will lead to more useful dashboards and fewer distractions caused by noise or incomplete data. ## What about row-level security? A common question about static-site generators is whether they can handle row-level security. The answer is yes, they do. Rather than just one package, the build process can create separate packages for each user or role. That way, when the request comes in, the server can then decide which package to serve them. This might seem to go against my earlier point about not having to run so many queries, but the build can be much more efficient with its queries than having to do it fully on demand for each user. The result is also still much faster for the user; and maybe your user permissions are by department or role, in which case multiple people still get served the same exact package. ## Summary A static site generator does not mean that the data is static, or even stale, or that the dashboards can’t be interactive. Quite the opposite. Because you can do so much more processing and Observable dashboards are built with code, you can enable many more ways to interact, explore, and model, than with traditional BI tools. Observable Framework makes for much faster dashboards, lower server load on your shared databases, and more useful dashboards with scheduled data updates. \]\]>https://observablehq.com/blog/align-dashboards-with-scales-of-business-decisions https://observablehq.com/blog/align-dashboards-with-scales-of-business-decisionsThu, 20 Jun 2024 00:00:00 GMTThe rate and resolution at which organizations collect and access data — like customer sign-ups, product usage, purchases, etc. — often differ from the scales of business decisions. A company might track website traffic minute-by-minute, but make decisions about site content and marketing campaigns based on trends observed over several weeks or months. Dashboards built to update at the speed of their raw data, instead of considering the scales of business decisions, can unnecessarily sacrifice performance and lose viewers in the weeds, leading to [dashboard rot](/content/blog/five-strategies-fight-dashboard-rot/index.html). In this post, we share strategies to help your organization build more effective data apps and dashboards that align with and clarify the requirements of business decisions. As a result, when stakeholders view your [data visualizations](/content/learn/what-is-data-visualization/index.html), they can focus on changes that actually matter to their organization. ## **Use scheduled builds for timely analytics** Real-time analytics in data apps allow decision makers to track metrics on the fly as new data comes in. There are cases where real-time analytics are required, for example if your team is monitoring wind speed and direction during a wildfire event to inform evacuation alerts. But real-time data analytics can come with costs for app performance and clarity. First, real-time analytics require data to be queried, transformed, and analyzed when a page is opened or refreshed. With big data and complex queries, that means users can find themselves watching a spinning wheel, waiting for charts to load while data is processed on the client. Waiting disincentivizes return viewers, reducing the long term use and value of your dashboard. Second, real-time analytics can — perhaps counterintuitively — make dashboards _less_ useful for decision-makers. As Observable founder and CEO Mike Bostock recently shared: > Business decisions often take time to effect. And in that case, it’s usually detrimental to make a business decision based on fast-moving data. If it’ll take 3–6 months to execute a new business strategy, you don’t want to base that decision on data that changes every minute! Mike Bostock This means that when dashboards pull in real-time data at a much higher frequency than the time scales of business decisions, the dashboards can become unnecessarily slow. This can lead viewers to focus on short term noise in the data instead of on meaningful longer term trends. So, in cases where real-time data access is not required, we recommend scheduling updates to keep your dashboards up-to-date _enough_ to meet viewers’ needs. With scheduled updates, you can automatically rebuild and redeploy your data app at a set time interval. (Learn how to set up continuous deployment using [automated builds](/content/framework/deploying#automated-deploys/index.html) with Observable Framework.) For example, our internal BI dashboard updates daily, which is frequent enough for our team to consider metrics like customer sign-ups, page traffic, and product usage. For other organizations, that might mean rebuilding every 30 minutes, hourly, weekly, quarterly, or another custom interval. There are two benefits of scheduled builds. First, because you’re not pulling in live data, you can push slow data processing to build time (e.g.with Framework’s [data loaders](/content/framework/loaders/index.html)), which means faster page loads for viewers. Second, scheduled updates can help viewers consider patterns over time scales more relevant to business decisions, with less focus on just the most recent observations. ## **Design charts and dashboards to align with scales of business decisions** Effective dashboards are opinionated, meaning that they are designed — ideally, with input from domain experts and expected users — to quickly guide viewers to the most important insights. The opposite is a dashboard that allows for (or even encourages) undirected [data exploration](/content/learn/what-is-data-exploration/index.html), which can leave viewers wandering through myriad data scenarios and struggling to focus on the most important changes or comparisons. Below, we share tips and strategies to create opinionated charts that align with scales of business decisions by smoothing, binning, and putting data in context. ### **Tip \#1: Smoothing can help viewers focus on trends, instead of noise** We’ve described how scheduled builds can help viewers focus on meaningful trends, instead of randomness in their data. Similarly, when displaying time series data, [moving averages](https://en.wikipedia.org/wiki/Moving_average) smooth patterns in the data, which can help to deemphasize outliers and noise. Adding a moving average is opinionated, because it guides a viewer to focus on a smoothed signal instead of noise in the raw data. Data visualization developers should opine further by choosing properties like the window size, alignment (e.g. if the window is centered or trailing), and which statistic is calculated for values in the window (e.g. a mean, median, or otherwise) for optimal smoothing. What “optimal” looks like will be data- and case-dependent, and likely requires input from stakeholders and domain experts. While moving averages are a form of aggregation, including them doesn’t require sacrificing richness in your data visualizations. For example, in our internal BI charts (created using a [window transform in Observable Plot](/content/plot/transforms/window#window-transform/index.html)), we include raw data as background (gray bars), while highlighting smoothed trends using both a 7- and 28-day moving average (blue and black lines, respectively). This layered approach helps a user focus on the forest, while still letting them see the trees. ### **Tip \#2: Bin, facet, or aggregate to match the scales of business decisions** Often, an organization’s data exists at a higher resolution than what’s needed for informed decision making. For example, an input form might gather data for customer age using their date of birth ( _i.e._, age can actually be determined in days), but messaging is crafted to target broader age bins (e.g. \[20 - 29 years\], \[30 - 39 years\], \[40 - 49 years\]). Presenting data at a much higher resolution than the scale of business decisions creates more work for viewers, because it adds an intermediate step between seeing a chart and deciding what (if anything) should happen in response. Put another way, the chart should lead directly to actionable insight, instead of requiring viewers to make the cognitive leap from the chart into determining how things fit into bins and then determining any insights. As a simple example, let’s consider country-level GDP and life expectancy data for 2022 from [the World Bank](https://www.worldbank.org/en/home). The scatterplot below explores relationships between country GDP and life expectancy, with the “Income group” variable mapped to fill color: There’s nothing inherently wrong with the chart above. But what if you know that the target dashboard viewers use the ordinal Income group variable, not continuous GDP, in their decision making? Does the chart above make it easy to compare life expectancy across Income groups? Consider the alternative below, which facets observations by Income group, and adds a horizontal rule at the mean life expectancy for each: The second chart shows the same data as the first, with each country represented by a dot. But in this version, faceting and the additional summary statistic help a viewer more quickly compare life expectancies across Income groups. Just as we can guide users to interpret their data at meaningful _time scales_ using scheduled updates and smoothing as described above, we can thoughtfully bin, aggregate, and facet data to help users focus on patterns that are most essential for their business decisions. In all cases, by customizing dashboards to align with scales most useful for the intended audience, we can cut the time and cognitive load they need to get from chart to decision. There are certainly times when you want to avoid any aggregation and maintain the maximum data resolution. For example, we were only able to detect some patterns in our web traffic because we could quickly plot over 7 million individual API requests, as described in our [Analyzing web logs example](https://observablehq.observablehq.cloud/framework-example-api/). ### **Tip \#3: Show scales relevant for business decisions directly** Business decisions are often made based on comparisons, for example with quarterly goals (e.g. “our sign-ups are currently at 78% of our Q2 goal”), or with a previous period (e.g. “revenue has increased by 21% this year, compared to 9% over the same period last year”). To help viewers interpret changes, dashboards should present data with context that clarifies the scale needed to make an informed decision. Consider the chart below, with mock data for email open rates for the current year (in green). Context is added in two ways: first, by adding a target open rate (yellow dashed line), and second, by showing values for the same time period from the previous year (gray line). The magnitude of the target open rate immediately helps a viewer to understand the scale on which they should consider the 2024 data. It is quick to see, for example, that despite recent declines, the 2024 open rate is still over the target, and just slightly below the rates recorded for the same dates in 2023. Well-designed dashboards avoid, whenever possible, making viewers look in multiple places to find the information they need to responsibly interpret data. By adding context (e.g. with meaningful thresholds, showing data for previous periods, and annotations) directly to a chart, viewers can quickly understand the scale on which they should consider their most recent data, with all relevant information in one place. ## **Start building better dashboards today with Observable Framework** By customizing dashboards and charts to align with the scales of business decisions, you can guide users to focus on changes that actually matter for their organization, so that they can make informed decisions, faster. When working in Observable Framework, everything is built with code. That means you can tailor each piece of a dashboard for your target viewers to put them on the fastest path from data to decisions. Create rich, fully customizable data visualizations using [D3](https://d3js.org/), [Observable Plot](/content/plot/index.html), or any other JavaScript library. If you’re new to Observable Framework, here are some resources to help you get started: - Check out [our examples](https://github.com/observablehq/framework/tree/main/examples#observable-framework-examples) to see what’s possible - Follow along with our [Getting started tutorial](/content/framework/getting-started/index.html) - Learn how to build the data visualizations described above, and much more, with [Observable Plot](/content/plot/index.html) \]\]>https://observablehq.com/blog/three-ways-build-better-dashboards-data-apps-with-code https://observablehq.com/blog/three-ways-build-better-dashboards-data-apps-with-codeTue, 11 Jun 2024 00:00:00 GMTIf you’ve ever used a traditional business intelligence tool, you’ve run into its limitations. There are only so many charts you can make in low-code tools before you need custom calculations. The kinds of data analysis you’re able to perform in these tools is limited to what the tool gives you, the layouts are often inflexible, and styling never quite looks right. Building dashboards using code lets you make your own choices and gives you the freedom to do almost anything you want. Here is a look at the kinds of things you can do with code, including processing and data modeling, pixel-perfect and flexible layouts, reusable components that save you time and effort, and the exciting world of unusual and custom visualizations. To see more examples of what is possible when you build dashboards with code, you might also want to check out [our Explore page](/content/explore/index.html). ## **Processing and data modeling** There are countless tutorials about how to do certain analyses using traditional [business intelligence](/content/learn/what-is-business-intelligence/index.html) (BI) tools. They usually consist of many unintuitive steps to end up with a cohort analysis, market basket analysis, or similar. You might get the analysis the tutorial lays out, but you won’t be able to adapt it to your needs, and next time you’ll have to do it all over again. These are complex procedures with many variations and nuances, which can’t easily be pre-packaged into a GUI tool. In a programming environment like Python or R, there are usually existing implementations that you can use, or modify if they don’t do exactly what you need. In addition, you can build entirely custom analyses or models using code. You might build them from scratch entirely, like you’d do in Excel, or you can use existing code components. R, for example, offers a plethora of statistical tools that is unmatched even in dedicated statistics packages (and certainly any BI tool), and Python has powerful data processing and manipulation tools as well. As a simple example, we have Observable Framework data loaders that [compute clusters using k-means clustering in R](https://observablehq.observablehq.cloud/framework-example-loader-r-to-csv/), or [logistic regression on a dataset in Python](https://github.com/observablehq/framework/tree/main/examples/penguin-classification), each with just a few lines of code. You can build anything here, including financial models, sophisticated machine learning processes like market-basket analysis, and more. In Observable Framework projects, [data loaders run at build time](/content/blog/data-loaders-for-the-win/index.html), so even complex and time-consuming operations on large datasets will not make the dashboard load any slower. ## **Precise layouts and stylings** A common complaint with traditional business intelligence tools is that it’s impossible to get the exact look you want. This is true both in terms of where elements are placed on the dashboard (“pixel-perfect layout”) and the entire design, such as color or branding. In traditional BI tools you may also end up with elements on your dashboard that you can’t change or remove. Making changes across elements (such as a subtle gradient across a section of the page) can also be close to impossible, instead having to change each one separately. You also usually can’t reuse your layouts and styles, which wastes times. Code, in the form of cascading style sheets (CSS), makes all of these things possible and often much easier than having to click through countless individual controls. CSS is hierarchical, so colors, spacings like margins or gutters, and typography can be set once and apply to the entire page. Specific elements can then be styled individually where they are different from the default. Perhaps some charts or big numbers need their own special backgrounds or a larger size for their titles. All that can be controlled very precisely. The layout of all of your dashboards’ elements is also under your full control, so of course your layout is pixel-perfect! And not only that, you can set it up so that it adapts to the size of the window or device you’re using. Elements can resize in different ways, perhaps charts scale more than filters or big numbers make way for other elements. The entire layout can change, so instead of a horizontal layout on a large screen you end up with a vertical stack in a narrow window or on a mobile device. Less important items can also collapse into little hamburger menus to make space. Mobile layouts for dashboards are still difficult and tedious in most traditional business intelligence software, but are built into the fabric of the web. They do require a certain level of setup and code to make use of, but once that is done they can be reused. Dashboards built with code can adapt to the size of the window or display not just by scaling, but also by changing their layout. Here are two views of [Observable Framework's mortgage rates example dashboard](https://observablehq.observablehq.cloud/framework-example-mortgage-rates/), with a third one further down. Note the different stacking of the cards between all three! ## **Reusable components to save time and effort** Dashboards and data apps often share items such as charts, data, as well as processing and modeling, etc. You certainly tend to want them to be built in a consistent way and follow the same style guides. And yet, with legacy BI tools, you need to build each new dashboard by hand, with little or no reuse of items. By building dashboards with code, you save time and effort by reusing existing components like charts, themes, or big number boxes from existing libraries or that you’ve built yourself. The plethora of statistics and analysis tools in R and Python mentioned earlier are widely used and vetted, which allows you to build useful analyses quickly, and with fewer sources for errors. Even within the same dashboard, you might be using the same kind of chart, or the same kind of analysis, for different data. With a code component, you can just call a function that performs the same task, such as producing the same big number with a chart and explanatory text created from the numbers. This is also true for layouts and all sorts of stylings that are usually reused for consistent  branding. Code (in the form CSS definitions) lets you create the look and change it in a consistent way, without having to worry about missing an item somewhere. Maintaining consistent styling and layout both within and between dashboards with the simple inclusion of a stylesheet saves time and effort. Building your own components and stylesheets is a great investment into your workflow. The up-front effort pays huge dividends whenever a new dashboard can just reuse it and show the familiar processing of the data while staying on brand. The two cards on this mortgage rates dashboard are created by the same function, which gets invoked twice for different data series. Note also the different layout compared to the earlier image, with the chart next to the boxes in the wider view. ## **Uncommon and custom data visualizations** Traditional business intelligence platforms are great for quickly putting together common chart types, but making truly custom charts and dashboards quickly becomes tedious. You might be able to coax a BI tool into showing you an unusual chart, such as a [Sankey diagram](/content/learn/glossary#sankey-diagram/index.html), but it will require a lot of work. Or you might be able to pick it from a chart gallery or marketplace, but you'll only be able to make limited changes. When you create data apps and dashboards using code, you can make use of the huge number of charting and visualization libraries, including those oriented towards [data exploration](/content/learn/what-is-data-exploration/index.html), like [Observable Plot](/content/plot/index.html) and those suited for bespoke interactive [data visualizations](/content/learn/what-is-data-visualization/index.html), like [D3](https://d3js.org/). This means that you can just reuse existing code examples to make more visualizations, including Marimekko charts, Sankey diagrams, dense yet interactive dot plots, node-link diagrams for graphs, custom maps, and more. Because it’s all code, the charts can be customized further to best display your data, and perhaps even come up with an entirely new way to show your data! Swoopy arrows, chord diagram, radar plot, ISOTYPE chart? These and many more uncommon chart types are available when building a data app or dashboard using code. ## **Conclusion** Code allows you to decide what to build. Code gives you control over your dashboards, both in terms of functionality and styling. Code makes it possible to reuse and transfer work, and make use of the vast number of libraries and code snippets out there. The best dashboards are built with code. \]\]>https://observablehq.com/blog/five-strategies-fight-dashboard-rot https://observablehq.com/blog/five-strategies-fight-dashboard-rotTue, 28 May 2024 00:00:00 GMTDashboards are incredibly important for understanding business intelligence, and yet a huge pain to maintain. They’re packed with valuable information, and yet they often fade from people’s attention when they don’t answer their questions. They’re critical for business operations, and yet end up unused by their intended audience once the novelty wears off. When [building dashboards](/content/blog/seven-ways-design-better-dashboards/index.html), a common problem that many teams face is dashboard rot — a situation where your dashboard becomes outdated, inaccurate, and ultimately ineffective for its users. The term is inspired by _bit rot_ in software (which needs to be kept updated or it will stop working properly) and _link rot_ (web pages disappear or change, so links to them end up going nowhere). A dashboard can quickly lose its value if it’s not regularly updated, maintained and monitored. Dashboards are built with the best intentions to answer a question, inform important decisions, and keep stakeholders informed. But after the initial excitement, they often end up being ignored. And yet, you can prevent dashboard rot if you know the reasons why it happens in the first place: - Dashboards are often built by default, even if a different display of data would be more effective. The wrong dashboard was built based on assumptions and unclear specifications. - The dashboard is static and doesn't allow interaction. - The data in the dashboard isn’t fresh. - The dashboard breaks and isn’t maintained along with its data sources. - The dashboard isn’t meeting a need, so stakeholders stop using it – and you’re not even aware of it. In this post, we’ll explore five key strategies to help you fight dashboard rot and keep your dashboards relevant and useful. ## Know whether to build a dashboard, and what to build > _“What do you mean, why’s it got to be built?” he said. “It’s a bypass. You’ve got to build bypasses.”_ > > – Douglas Adams, The Hitchhiker's Guide to the Galaxy When there is a question about business data, building a dashboard often seems like the obvious response to that question. But it’s worth taking a step back and asking if a dashboard is the best solution. Hastily built dashboards that answer a single question often end up rotting away in obscurity once they’ve done their job. Perhaps the question can be answered differently, with a simple database query, a script, or maybe [an Observable notebook](/content/documentation/notebooks/index.html)? A quick answer to a simple question can be perfectly sufficient and avoids spending more time than necessary. When the data in question is likely to be of interest to more people, and on an ongoing basis, a dashboard is a good choice. Now the question shifts to knowing what to build. This might seem obvious, but the excitement about building something new can cloud our ideas of what we really need. Make sure to ask the right questions: - Where is the request coming from, and who will the users be? The sponsor of a dashboard and its users can be the same, but they often are different people or departments, with different questions and needs. - What tasks are stakeholders hoping to accomplish with this new dashboard? What information are they looking for? Do you know what data sources provide the right data for the dashboard? - Are all the stakeholders involved in coming up with requirements and getting their voices heard? Are you making assumptions about their needs? - What existing tools and dashboards do stakeholders have, and why are those not sufficient? - Does the information your audience want shown in the dashboard match the tasks? Building a dashboard or data app without knowing all the requirements, or making the wrong assumptions about user needs, is perhaps the most common cause of dashboard rot. The time spent upfront to make sure you know your users’ needs pays off later with a better user experience and less toil modifying your new dashboard. ## Add interaction to improve data visualizations Static dashboards can be useful, but nothing beats allowing users to modify what is shown. In addition to engaging users, [interactive dashboards](/content/blog/how-to-add-interactivity-observable-framework-dashboard/index.html) are less likely to succumb to dashboard rot because users can more easily access the information they need. Filters allow users to modify the dashboard content by drilling down into a specific subset of the data. Mouse-overs provide additional information on charts and selections can highlight corresponding elements across the dashboard. Sometimes users need to access additional context. For this use case, links can point them to more information elsewhere, whether it's another dashboard or a different destination. [Observable provides a library of inputs](/content/framework/lib/inputs/index.html) to use in Observable Framework projects, including drop-down boxes and radio buttons for categorical data selections, rangle sliders to change continuous values, buttons, and more. ## Ensure data in your dashboard is up-to-date Dashboards are all about current data, but what is current depends on the use case. So it’s important to ask: what does your specific use case actually require? Is it the data as of close of business the day before? The end of the previous month or quarter? Knowing, and making a conscious decision, about what constitutes [current data for your dashboard](/content/blog/align-dashboards-with-scales-of-business-decisions/index.html) helps you avoid incomplete time periods that can cause confusion. Incomplete data is surprisingly common on dashboards, and can lead people to draw wrong conclusions even when they’re aware of the problem. For example, if a line for sales is pointing down, it may be because sales are actually down, or it might be because you’re looking at an incomplete month’s worth of data. A clear and intentional strategy for updating the data avoids this problem by always showing data for complete time periods that can properly inform decisions. Scheduled builds are a great way to ensure your data is current as of a particular date and time. [Observable Framework](/content/framework/index.html) supports [automated deploys](/content/framework/deploying#automated-deploys/index.html) to set up continuous deployment (or CD). Traditional [business intelligence](/content/learn/what-is-business-intelligence/index.html) (BI) tools can achieve the same thing with date filters that remove incomplete time periods, though they still need to load all the data when the dashboard loads. ## Maintain your dashboard and update data sources Not only the data needs to be kept up to date, the dashboard itself might, too! Check back with your users to see if they’re still getting value out of it, or if their needs have changed (and track their usage!). When people start using something, they often realize that they actually needed something different after all. Dashboards require access to data sources, which might change or disappear. Dashboards can break as a result, so it’s important to keep an eye on them and be ready to respond when that happens. Even better, have a plan for any changes to databases to be run by the BI team so they know ahead of time when a change is coming, and can evaluate the impact it will have on them. A separate build process, like the one in [Observable Framework](/content/framework/index.html), helps with this since it catches errors in one central place. The deployed dashboard still works, but the build process can send error messages to the maintainer, who can fix problems. The user might never even know that there was a problem, while in a traditional BI dashboard, they will see database errors without knowing what to do about them. ## Share your dashboard and track dashboard usage One way to know if a dashboard actually delivers value to your end users is to track dashboard usage. It’s nice to get validation that people actually find what you’ve built useful. But more importantly, you want to make sure if it gets used at all, and used as much as you’d expect, over the long term. Dashboards often get a spike in traffic when they’ve just been released, but then the traffic drops off. _Build it and they will come_ – except, that doesn’t usually work. You have to tell people that you made something for them, and keep reminding them. Perhaps your immediate customers are aware of it, but are there other stakeholders or interested parties who might benefit from your dashboard (who also have access to the data, of course)? [Observable’s project analytics](/content/blog/how-to-track-dashboards-enhanced-analytics/index.html) show you important information about how your dashboards and data apps are being seen, and who your most enthusiastic (or at least, most frequent) users are. A dashboard that doesn’t get used is no better than not having the dashboard at all – or perhaps even worse, since you spent all this time building it! ## Conclusions Dashboard rot is real, but it can be avoided with some simple strategies. Dashboards sometimes get a bad rep because of the issues discussed above: it didn’t need to be a dashboard, it was built without enough information about the what and why, etc. But none of these problems are insurmountable. With some forethought and the right tools (such as project analytics), the work you do building a dashboard or data app will provide lasting value to your users. By implementing these strategies when building your next dashboard, you can ensure that your dashboards remain relevant, accurate, and useful over time. Regular updates, user feedback, and performance improvements will help prevent dashboard rot, and ensure your dashboards continue to provide valuable insights over the long term. Want to learn more about how Observable’s end-to-end data visualization platform can help you quickly build fast, interactive dashboards help your end users make more informed decisions? Start [building with Observable](/content/pricing/index.html) today. ## Frequently asked questions ### What is dashboard rot and how does it affect my dashboards? Dashboard rot occurs when a dashboard becomes outdated, inaccurate, or less effective over time due to lack of maintenance or updates. It can lead to users relying on incorrect or irrelevant data, reducing the dashboard's usefulness and value for decision-making. ### How can I prevent dashboard rot when building dashboards? By maintaining your dashboard’s data sources, ensuring data is up-to-date, elevating your dashboard’s user experience through interactivity, and monitoring dashboard usage, you can ensure your dashboard remains effective. ### How often should I update the data in my dashboard? You should update your dashboard's data and visualizations regularly to ensure it remains accurate and relevant. The update frequency depends on your organization, but it's important to set up automated refreshes or scheduled updates to avoid outdated or incomplete data. \]\]>https://observablehq.com/blog/write-troubleshoot-data-loaders https://observablehq.com/blog/write-troubleshoot-data-loadersThu, 16 May 2024 00:00:00 GMTWhen working in [Observable Framework](/content/framework/index.html), you can build faster, richer data apps by precomputing data snapshots at build time with [data loaders](/content/framework/loaders/index.html). Data loaders can be written in any language (Python, Rust, Julia, SQL, R, _etc._) as long as they send their payload (usually a data snapshot formatted as a CSV, JSON, image, parquet, ZIP archive, _etc._) to [standard output](https://en.wikipedia.org/wiki/Standard_streams#Standard_output_(stdout)). The process of writing data loaders might feel a bit unfamiliar at first. For example, how can you see what’s generated by a data loader as you work? Where should you look for clues when something goes wrong? And how can you set yourself up for success? In this post Observable team members share useful tips to help you write and troubleshoot data loaders. Let’s dive in! ## **Tips for writing data loaders** ### **Embrace the flexibility: choose** **_your_** **tools** Data loaders can be written in any language and with whatever libraries you choose _._ We encourage you to embrace data loader flexibility, and **write loaders in the language you feel most confident using for the current task.** Fil Rivière (Data Visualization Engineer) shares: > \[A data loader is\] just a program. Any program. You can use anything that runs on your computer. Fil Rivière Observable Data Visualization Engineer And, as Michael Cooper (Senior Software Engineer) highlights, that means you can choose to write them with familiar tools: > One of the things that’s important is you shouldn’t be intimidated. \[Data loaders\] are normal programs. It should be something very familiar — data loaders are designed to meet you where you are. Michael Cooper Observable Senior Software Engineer So get started on the right foot when you set out to create a data loader. Choose the language and libraries you’re comfortable with to set yourself up for success. ### **Run the data loader** This one might seem obvious, but we sometimes overlook obvious strategies when getting started with a new workflow. So remember: you can **manually run data loaders** to get a quick view of the output. Since the generated file is sent to standard output, you’ll see the printed output in the terminal for most formats. Mike Bostock (Founder, Office of the CEO) recently responded to a question about troubleshooting data loaders: > Data loaders are just programs, so you can run them manually. For example you can run a JavaScript data loader as node src/foo.csv.js. Mike Bostock Cobus Theunissen (Head of Customer Success) notes: > One of the things that I found tricky was running a TypeScript data loader, since it needs to be compiled to JavaScript in order for node to run it. I just use npx tsx src/data/loader.xxx.ts, which compiles the TypeScript on the fly. Cobus Theunissen Let’s look at an example. The sample Julia data loader below `(src/data/chinstrap.json.jl)` generates a JSON file: ``` using CSV using DataFrames using JSON3 chinstraps = CSV.read("src/data/penguins.csv", DataFrame) |>              filter(:species => ==("Chinstrap")) chinstrapsJSON = JSON3.write(chinstraps) println(chinstrapsJSON) ``` When the loader is run — either from the Julia REPL using `include("src/data/chinstrap.json.jl")`, or from the command line with `julia src/data/chinstrap.json.jl`) — we see the following printed (only the first line is shown for brevity): `{"columns":[["Chinstrap","Chinstrap","Chinstrap","Chinstrap","Chinstrap",...`\ \ Don’t expect this to be the cleanest view of your data snapshot. Michael notes that “When you run the data loader as a script, sometimes it might dump out a lot of information if it’s working.” Despite the display, it does quickly confirm some important things: the loader is sending _something_ to standard output, and it looks like data in JSON format. Both are good clues that we’re on the right track.\ \ There are some output types (e.g. ZIP archives and images) that won’t display in the terminal. In those cases, you can send data loader standard output to a file, e.g. with node `src/foo.json.js > foo.json`, then view the file with any program that can open it.\ \ ### **Display generously in live preview**\ \ Changes to data loaders are reflected in [live preview](/content/framework/getting-started#test-live-preview/index.html). That means you can display code and content as you iterate to check data loader outputs.\ \ The sample data loader below (`magic.json.js`) generates a JSON file with information about Magic: The Gathering cards from [scryfall.com](http://scryfall.com/):\ \ ```\ import * as d3 from "d3";\ \ const url = "https://api.scryfall.com/cards/search?order=cmc&q=c:red%20pow=3";\ \ const magicCards = await d3.json(url);\ \ const magicCardsData = magicCards.data.map((d) => ({\   name: d.name,\   release: d.released_at,\   mana_cost: d.mana_cost,\   set: d.set_name,\   rarity: d.rarity\ }));\ \ process.stdout.write(JSON.stringify(magicCardsData));\ ```\ \ `FileAttachment()` below accesses the output, magic.json, in a Markdown file, and `display` exposes the value. The `echo` option additionally exposes the code, which can be helpful during development:\ \ ````\ ```js echo\ const magicCards = await FileAttachment("data/magic.json").json();\ \ display(magicCards);\ ```\ ````\ \ \ \ We can view the output in a nice table with [`Inputs.table()`](/content/framework/inputs/table/index.html):\ \ ````\ ```js echo\ display(Inputs.table(magicCards));\ ```\ ````\ \ \ \ ### **First aim for “it works,”** **_then_** **optimize**\ \ Drafting data loaders can feel like a bit of a “chicken or the egg” scenario. Egg: you need to write a loader to get a data snapshot from which you can start developing visualizations. Chicken: data loaders should be tailored to only generate data that gets used in visualizations. But how do you know the minimum data snapshots you need until you’ve prototyped the visualizations?\ \ [Paul Buffa](https://www.linkedin.com/in/paul-buffa/) (Director of Business Intelligence) advises:\ \ > Start simple and don’t focus too much on optimizing the data loader at first. The visualization drives the data loader, so first you need to know what the visualization needs. Then, you can update the loader to optimize build time — but that doesn’t need to be your first step.\ \ \ \ Paul Buffa\ \ Early versions of a data loader will almost always send more than you need to the client. That’s okay. Get your draft data loader up and running, get a better sense of what your visualizations require, _then_ focus on optimizing a data loader to minimize what gets loaded by the page.\ \ ### **For data loaders and client-side queries, we** [**❤️**](https://emojipedia.org/red-heart) **DuckDB**\ \ Several Observable team members mentioned how writing SQL queries using DuckDB improves their data loader performance and development experience. In Framework, you can use DuckDB for fast data loaders and SQL queries in Markdown files.\ \ Related to his tip about “get it working first, then optimize” above, Paul shares this strategy for quick iteration, recommending client-side SQL queries in Markdown for faster development:\ \ > You don’t have to pull straight from the warehouse every time. Just pull in more with the data loader in a rawer format, then use SQL code blocks with DuckDB in Markdown to query and iterate more quickly.\ \ \ \ Paul Buffa\ \ Then, you can easily adapt queries you’ve honed in a Markdown file to optimize a DuckDB-powered data loader.\ \ Fil likes DuckDB for flexible, efficient data loaders and client-side queries:\ \ > DuckDB is a rising star, which you should use for data loaders. It digests any data source, and is very efficient at every level.\ \ \ \ Fil Rivière\ \ Data Visualization Engineer\ \ So give DuckDB a shot! Learn more about [SQL in Markdown](/content/framework/sql/index.html), and see an [example data loader](https://observablehq.observablehq.cloud/data-loader-examples/shell#parquet) powered by DuckDB that emits a parquet file.\ \ ## **Tips for troubleshooting data loaders**\ \ ### **See logged error messages**\ \ When writing data loaders, there are different ways to see logged messages and errors.\ \ First, check for errors returned by the preview server. The preview server watches data loaders and automatically re-runs them when changes are made. Errors thrown during execution are shown in the **preview server log**.\ \ Below, an error returned for a Python data loader (where the variable name should be `Birth_Rate`, but is incorrectly written as `Birth_rate`) is shown in Framework’s preview server log:\ \ \ \ If you’ve tried to access the output of a data loader on a page, you will also see an error returned there, typically like:\ \ `RuntimeError: Unable to load file: birth-statistics.png`\ \ You can create your own log to see what’s returned at useful checkpoints in your data loader. Remember to log messages to [standard error](https://en.wikipedia.org/wiki/Standard_streams#Standard_error_(stderr)), not standard output, so that generated messages aren’t included in the output file. In JavaScript data loaders, for example, use `console.warn` to log to standard error (rather than `console.log`, which writes to standard output).\ \ ### **Check that everything is installed**\ \ When you write data loaders, any **modules, packages, and libraries used in the loader must be installed.**\ \ If you’re getting an error about modules or libraries not existing in the preview server log (like `code: ‘MODULE NOT FOUND’`), make sure you’ve [installed everything you’re using in a data loader — in the right place](/content/framework/loaders#execution/index.html) ( _e.g._ if using a virtual environment).\ \ Another thing to keep in mind: Framework’s [recommended libraries](/content/framework/imports#implicit-imports/index.html) are available out-of-the-box in Markdown files, but they are not implicitly imported in data loaders. For example, you can use D3 functions in JavaScript cells in Markdown without importing D3, but you’ll need to explicitly importit for use in a JavaScript data loader, _e.g._ with:\ \ `import * as d3 from "d3";`\ \ ### **Explore the cache**\ \ Files generated by data loaders are stored in the cache at `src/.observablehq/cache`. That means you can **view the files in the cache** to explore the outputs.\ \ If you’re struggling to load the output of a data loader in a page, Fil recommends:\ \ > Try opening the file from the cache. If you expect a JSON and it doesn’t open, just open the file in a text editor and see what it spits out.\ \ \ \ Fil Rivière\ \ That can be useful for debugging. For example, if you’re expecting to see data in JSON format but you instead see a series of warning messages, you might check to see if you’ve logged messages to standard output instead of standard error.\ \ Cached file names provide an additional clue. If there is an error during data loader execution, the generated file ends with a numeric identifier and `.err` suffix.\ \ The cache below contains three files generated by functional data loaders (birth-statistics.png, chinstrap.json, and magic.json), and two (birth-statistics.png.2015.err, birth-statistics.png.6045.err) from data loaders that exited due to error:\ \ \ \ You can force a data loader to re-run during development or on build by deleting the generated output. For example, on Unix machines you can remove entire observablehq/cache directory with rm `-rf src/.observablehq/cache`. Or, include the file name to remove the output for a single data loader ( _e.g._`rm -f src/.observablehq/cache/chinstrap.json`).\ \ ## **Start writing data loaders**\ \ We hope these strategies encourage you to take the plunge into writing and troubleshooting data loaders. If you’re new to Observable Framework, we recommend following our [Getting started tutorial](/content/framework/getting-started/index.html), which includes a friendly step-by-step introduction to creating and editing your first JavaScript data loader.\ \ Then check out our [Framework documentation](/content/framework/index.html) to keep building fast, rich data apps with code.\ \ \]\]>https://observablehq.com/blog/data-loader-examples-r-python-shell-scripts-julia https://observablehq.com/blog/data-loader-examples-r-python-shell-scripts-juliaTue, 30 Apr 2024 00:00:00 GMT[Observable Framework](/content/framework/index.html) uses JavaScript for code running in the browser, but the equally (or more?) important data loaders can use any language. Don’t believe us? Well, here are examples of using Python, R, shell scripts, and Julia. Even if you don’t regularly use JavaScript, you can use your skills in other languages to build fast and beautiful data apps and dashboards. You can walk through the code here, then [clone the examples repo](https://github.com/observablehq/data-loader-examples) (created by my colleague Allison Horst) yourself to build on them. The examples assume that you have the respective runtimes and libraries installed. ## **How do data loaders work?** When you build a Framework project, [data loaders are run](/content/framework/loaders/index.html) to fetch data from sources like databases and APIs, or run models. Their output goes into data files that are usually data formats like CSV, parquet, etc., but can be anything, like images. They are then packaged up and [deployed to the server](/content/framework/deploying/index.html). To learn more about data loaders,  see [our recent blog post](/content/blog/data-loaders-for-the-win/index.html). When your data app loads, these files are loaded with the page, which is much faster than querying a database. This is even more true when running analysis or models on the data. It’s also powerful, because you can run code in whatever language you like. That also gives you access to existing libraries such as pandas in Python, the vast selection of statistics and machine learning tools in R, etc. But no matter what you use on the backend, the resulting data app runs natively in your browser, and is fast and responsive. ## **Using data loaders for logistic regression in Python** First, let’s try a simple logistic regression on a dataset. This data loader, written in Python, loads in the [Palmer Penguins dataset](https://allisonhorst.github.io/palmerpenguins/) using [pandas](https://pandas.pydata.org/), and then computes the logistic regression between the penguins’ body measurements and their species. It then adds a predicted\_species column to the dataset and exports it as a CSV file. ``` # Import libraries (must be installed) import pandas as pd from sklearn.linear_model import LogisticRegression import sys # Data access, wrangling and analysis df = pd.read_csv("docs/data-files/penguins.csv") df_complete = df.dropna( subset=["culmen_length_mm", "culmen_depth_mm", "flipper_length_mm", "body_mass_g"] ) X = df_complete.iloc[:, [2, 3, 4, 5]] Y = df_complete.iloc[:, 0] logreg = LogisticRegression() logreg.fit(X, Y) results = df_complete.copy() results["predicted_species"] = logreg.predict(X) df_out = df.merge( results[["predicted_species"]], how="left", left_index=True, right_index=True ) # Write the data frame to CSV, and to standard output df_out.to_csv(sys.stdout) ``` The original code lives [in this repository here](https://github.com/observablehq/data-loader-examples/blob/main/docs/data/penguin-logistic.csv.py). ## **Fetching data to create a parquet file with DuckDB in a shell script** If you just need to run a command, perhaps to pull a file from a URL or copy from somewhere else, a simple shell script might be the way to go. Or you could script [DuckDB](https://duckdb.org/) to do the work for you and also run it from a shell script, as we do in this example. We could have also used curl here, but DuckDB can load files directly from a URL. In this case, we’re loading data about fuel stations from an OpenEI.org API. We then select only the fuel stations in California, pick just the columns we want, and finally export the data as an efficient [parquet file](https://parquet.apache.org/). This can then be [loaded into DuckDB](/content/framework/lib/duckdb/index.html) again inside the browser, or directly used with [Observable Plot](/content/plot/index.html) to create visualizations. ``` duckdb -csv :memory: << EOF CREATE TABLE allp AS ( FROM 'https://data.openei.org/files/106/alt_fuel_stations%20%28Jul%2029%202021%29.csv' ); CREATE TABLE cafuelstations AS ( SELECT "Fuel Type Code" as Type, State, ZIP, Latitude, Longitude FROM allp WHERE State = 'CA' ); COPY cafuelstations TO '$TMPDIR/cafuelstations.parquet' (FORMAT 'parquet', COMPRESSION 'GZIP'); EOF # isatty if [ -t 1 ]; then echo parquet file output at: $TMPDIR/cafuelstations.parquet echo "duckdb -csv :memory: \"SELECT * FROM '$TMPDIR/cafuelstations.parquet'\"" else cat $TMPDIR/cafuelstations.parquet rm $TMPDIR/cafuelstations.parquet fi ``` See [the original data loader here](https://github.com/observablehq/data-loader-examples/blob/main/docs/data/alt-fuel-stations.parquet.sh). ## **K-means clustering in R using a data loader** How about some simple clustering using the built-in features in R? This data loader reads in the same penguins dataset as the Python example above, converts it into a data frame, and then runs k-means clustering on it. The result is exported as a new CSV file, to be used on a dashboard or other data app. ``` # Attach libraries (must be installed) library(readr) library(dplyr) library(tidyr) # Data access, wrangling and analysis penguins <- read_csv("docs/data-files/penguins.csv") |> drop_na(culmen_depth_mm, culmen_length_mm) penguin_kmeans <- penguins |> select(culmen_depth_mm, culmen_length_mm) |> scale() |> kmeans(centers = 3) penguin_clusters <- penguins |> mutate(cluster = penguin_kmeans$cluster) # Convert data frame to delimited string, then write to standard output cat(format_csv(penguin_clusters)) ``` If you want to try it yourself, [you can find the source here](https://github.com/observablehq/data-loader-examples/blob/main/docs/data/penguin-kmeans.csv.R). ## **Processing text with Julia** Our final data loader example is written in Julia, and it shows one of the more unusual data files: text. This little example script fetches a book from Project Gutenberg, parses the text into paragraphs, and then pulls out one particular paragraph. You could easily build more on this, like text mining, sentiment analysis, and more. ``` #!/usr/bin/env julia # Load Julia packages (must be installed) using HTTP using Gumbo using TextAnalysis # Function to fetch text function fetch_text_from_url(url::String) response = HTTP.get(url) text = String(response.body) text = replace(text, "\r" => "") return text end # Split into paragraphs function split_into_paragraphs(text::String) paragraphs = split(text, "\n\n") return paragraphs end # Return a paragraph by number function get_paragraph_by_number(text::String, paragraph_number::Int) paragraphs = split_into_paragraphs(text) return paragraphs[paragraph_number] end # Text URL url = "https://www.gutenberg.org/cache/epub/1065/pg1065.txt" # Fetch text and access a paragraph by number text = fetch_text_from_url(url) paragraph_number = 29 result_paragraph = get_paragraph_by_number(text, paragraph_number) # Print text to standard output println(result_paragraph) ``` [The code is available here](https://github.com/observablehq/data-loader-examples/blob/main/docs/data/raven.txt.exe) for your perusal. ## **Literally any other language** The examples above are just to show a few specific pieces of code. Observable Framework recognizes a few file extensions by default, but you can run code in any language. One way to do this is to use a shell script as a bridge, as you saw with the DuckDB example above. If you want to make things more official, you can [register your own extensions and associated interpreters](/content/framework/config#interpreters/index.html). As of this writing, the list of languages that are supported by default includes JavaScript, TypeScript, Python, R, Rust, Go, Java, Julia, PHP, shell scripts, and even binaries. Go ahead, and try it yourself in a Framework project, [it’s easy to get started](/content/framework/getting-started/index.html)! Whatever process or pipeline you want to use to access, process, or model your data is available now, and can power fast, interactive data apps that don’t make your users wait. \]\]>https://observablehq.com/blog/how-to-track-dashboards-enhanced-analytics https://observablehq.com/blog/how-to-track-dashboards-enhanced-analyticsWed, 17 Apr 2024 00:00:00 GMTIf a dashboard is published and no one looks at it, can it be effective? The best type of dashboard is one your team actually uses. If you ask your users directly, it might sound like they are frequent visitors of your work, but it's hard to be certain. Maybe they’re supposed to look at your dashboards everyday but they forget. Maybe they’re still using those old Excel spreadsheets. _You can’t manage what you can’t measure._ If you can’t track how much your dashboard is being used, you can’t know whether it’s effective. _\[...Observable enters stage right\]_ We’re pleased to announce our newest addition to Observable: [**project analytics**](/content/documentation/projects/analytics/index.html). When you deploy a Framework project to Observable, project analytics enables you to see how much a project is being viewed and by whom. It's a great way to understand how effective your dashboard is. Are users checking in once a week or every day? Who are the regulars to certain pages and views? Is my work gaining or losing traction? You no longer need to stay up all night pondering life's great mysteries like “is my boss actually looking at my KPI dashboard??” When you log into your Observable account, you’ll see a new section in deployed projects showing the daily viewers to your dashboards. This example is from a project that I manage for our internal [business intelligence](/content/learn/what-is-business-intelligence/index.html), aptly named Observable BI. This visualization here is a sparkline-esque snapshot of viewer activity to your project. Each rectangle represents viewer activity for the past 60 days, with the darker blue colors indicating a larger number of viewers and light yellow indicating fewer viewers. This is only a taste of your project analytics, a bite-sized signal that you can scan across projects. To see your project analytics in all its glory, click the gear icon in the top right corner. Here we are provided with a richer view of our data. The first row shows viewer stats for all the pages in a project while subsequent rows are for a specific page. The daily viewer numbers (with a helpful x-axis at the top) reveal usage patterns, so I can see things like day-of-week trends (i.e. more viewers during weekdays / fewer on weekends). To the right of these usage pattern visuals, there is a total count of viewers for that specific page over 60 days. My favorite section in this report is the Regulars column. It shows me anyone on the team who has viewed this page at least two times a week. Here are the results for the marketing dashboard: I love seeing that the members of the marketing team are regulars to the marketing dashboard and they check this page, on average, four times a week. (I know what you’re thinking – only four days a week??) Joking aside, it’s really encouraging for me, the creator of these dashboards, to know that the right people are viewing them and doing so regularly. Seeing the regulars is also a great way to notice when a dashboard is not hitting the mark. I noticed that Hanna, our Head of Marketing, wasn’t a regular visitor to one of the pages I had created for her team. I reached out to her, we had a productive conversation, and I was able to quickly update the dashboard to accommodate her requests. Now I spot her as a regular every week! ## Collaboration is key At Observable, we strongly believe that the best dashboards are built with code, because it’s in the customization where you can craft a story with your data. The dashboard above was achieved through a collaboration with data visualization developers and designers on the team, and that is what makes it so effective. On my own, I would have just built a simple line chart or bar chart showing the daily viewer numbers. While this [data visualization](/content/learn/what-is-data-visualization/index.html) is somewhat interesting, it lacks the nuance that tells a story. One of my favorite such details is the user avatars in the Regulars column so you can quickly scan who the top viewers are across pages. (Yes, I'm the pizza avatar, if you hadn’t already guessed.) I could never see that with a bar chart view. I used to joke that the only way I knew people were using my dashboards was if people were complaining about them. With project analytics, I now have a much more positive way of knowing how effective my work is. If you’re an existing Observable user, you can view analytics for all of your projects that are hosted on Observable by logging into your workspace. If you haven’t yet tried Observable Framework (or haven’t yet deployed a project to Observable), what are you waiting for? You can learn more about how to [get started in our docs](/content/framework/index.html). \]\]>https://observablehq.com/blog/crafting-data-colors https://observablehq.com/blog/crafting-data-colorsTue, 09 Apr 2024 00:00:00 GMTColor is a cornerstone of nearly every dashboard and [data visualization](/content/learn/what-is-data-visualization/index.html). A well-designed data visualization color palette enhances readability, ensures accessibility, and reinforces brand identity. Whether you’re designing dashboards, reports, or interactive charts, selecting the right color palette can transform raw data into a compelling visual story. However, when incorrectly used, color can muddle your communication or dilute your brand. When the same shade of blue means “Eastern region” in one chart and “service interruptions” in another, viewers need to work harder to read a chart or dashboard, or they might simply misinterpret the data. Given the important role that color plays in communicating effectively through data visualizations, we designed [Observable Plot](/content/plot/index.html)’s default categorical colors to create rich, easily-interpretable displays of data. The Observable 10 palette helps speed creation of effective, rich data displays by offering a breadth of colors. It’s the new default scheme for categorical color in Observable Plot, our open-source JavaScript library for exploratory data visualization. Previously, Plot used [Tableau 10](https://www.tableau.com/blog/colors-upgrade-tableau-10-56782) as the default categorical palette. While Tableau 10 is effective, we wanted to lead with a contemporary [color scheme](/content/@d3/color-schemes/index.html) that matched the creative spirit of our open source, expressive framework. The new Observable 10 color palette accomplishes this goal and better expresses our brand. In the post below, we discuss how we designed an effective default color palette for data visualizations for Observable Plot, and showcase how these steps apply to identifying a unique color scheme for your organization. ## **What are the principals of designing a categorical color scheme for data visualizations?** When we set out to make a new color scheme, we first identified some universal principles. We wanted a color palette that: 1. Offers as many colors as possible to support the creation of rich, multidimensional data visualizations. One chart might not need ten colors, but dashboards and other collections will. 2. Connects to an established color palette. For Observable 10, we wanted to express the creativity and connection embodied in our products and, by extension, our brand identity. 3. Supports differentiation among the colors even in small or thin marks. Big, adjacent blocks of color are easy to distinguish; it is important that two small dots on the opposite sides of a scatter plot would read clearly, too. 4. Offers reasonable differentiation for people with color vision deficiency. When we designed our color palette for data visualizations, we set out to deliver a broadly accessible and inclusive solution. 5. Uses colors that easily map into common names like “blue” and “orange.” Conversations around data displays work better when there is clear shared terminology instead of questions like, “Are you talking about the blue-green dot or the green-blue one?” 6. Works on both light and dark backgrounds. As an exploratory and expressive data visualization platform, Observable’s default colors should work regardless of your vibe. ## **Building a new color palette for data visualizations** As with any creative endeavor, the path to the Observable 10 was one of iteration. There was also more than a little trial and error! The first principle — offer as many colors as possible — pushed us to have lots of colors in the finished palette, but that isn’t much of a starting place. For that, we turned to our brand guidelines, which have these seven core colors. It’s easy to see that Faint Blue and Washed Yellow would not work on light backgrounds, so we tossed those out. To see how the remaining 5 colors would work together, we followed in the footsteps of Maureen Stone and Cristy Miller when they developed the Tableau 10. We built an [Observable notebook](/content/@observablehq/categorical-palette-tool/index.html) that let us see the colors in two side-by-side plots as well as in text and common mark shapes. These plots show the colors in [CIELAB](https://en.wikipedia.org/wiki/CIELAB_color_space) (or sometimes just “LAB”) [color space](https://ericportis.com/posts/2024/okay-color-spaces/). Unlike RGB or HSV, LAB covers the entire gamut of human visual perception, and it does it along three dimensions: a\* (pronounced “a-star”), b\*, and L\*. a\* is a spectrum between green and red, while b\* runs along blue and yellow. L\*, for lightness, goes from black (0%) at one end to white (100%) at the other. Here’s what the a\*-b\* scatter looks like with a fixed lightness of 70%. In this scatter plot you might notice that saturation is lowest at the center where it is gray and highest at the periphery. Hue — the shade of color — varies with the angle. This demonstrates how LAB mirrors typical human perception, which sees many different shades of green and blue from 6 o’clock to 10 o’clock, while we can differentiate very few different shades of yellow around 12 o’clock. Looking at our starting palette in the visualization above, we can see that the 5 colors vary significantly with their angles, so it’s a promising start. However, Mint Chip and Goldenrod are very close to each other on the L\* lightness scale, and we’ll want to keep an eye on that. ### **Adding more colors to our data visualization color palette** To fulfill our primary goal, we needed a scheme with more than five colors, so we added five more. We kept the blue, purple, and red (formerly Coral) as they were. We dropped Mint Chip in favor of two colors: a cyan a little closer to blue (3) and a true green (8). Orange (1) became a little less saturated and lighter. We also added a cool gray (9) right near the center of the scatter, pink (5), and brown (6). The final addition in this draft of the palette was light blue (7). While LAB is great for reviewing the colors in a palette, finding nearby colors can be challenging. RGB is not much help either. To identify candidate colors, it helps to work in a color space like OKLCH, where LCH stands for lightness, chroma, and hue. These three dimensions map more naturally to how we think about color, and the [OKLCH color picker](https://oklch.com/#54.8,0.165,265.62,100) makes it easy to move along them. ### **Designing for accessibility in data visualizations and dashboards** Our fourth goal was to create a palette that could work reasonably well for people with color vision deficiency (CVD). The most common CVD is deuteranopia, a lack of red-green differentiation that affects about 4% of people. Using a tool like [Color Oracle](https://colororacle.org/), we can simulate what the palette looks like for people with this form of CVD. Among 10 colors, most of them are distinguishable, but the cyan (3) and pink (5) are very close. While the two colors have very different a\* values, they are similar on both b\* and L\*, which is why they are problematic for people with deuteranopia. However, moving either of those colors along b\* moves them too close to other hues, creating another differentiation problem. We left the cyan and pink where they were. Visual accessibility can also be improved by encoding data in other ways, either redundantly with color or as a replacement. Other choices include labeling with [text](/content/plot/marks/text/index.html), using different [shapes](/content/plot/marks/dot/index.html) for different categories, adding hover labels with [tips](/content/plot/marks/tip/index.html), varying thickness or dash patterns in [lines](/content/plot/marks/line/index.html), or even changing to using [bars](/content/plot/marks/bar/index.html) or another positional mapping. ### **Crafting color palettes that work for light and dark mode** The final pass over the Observable 10 palette was to review against dark and light backgrounds. To do this, we applied the palette against a test suite of dozens of charts built with Observable Plot. We noticed two particular issues — on light backgrounds the orange (1) was not dark enough, and on dark backgrounds the green (8) was too vibrant, almost like it was buzzing. To address these problems, we experimented with other nearby colors. Ultimately, we chose a more saturated orange, moving it farther away from the center, and a darker green farther down on the L\* plot. ### **Ordering colors for differentiation in data visualizations** With all the colors in place, the last piece of the puzzle was choosing their order since we want charts and data visualizations with fewer than ten colors to look and feel great, too. The first two colors in the scheme are blue and orange. Their high difference in both a\* and L\* make them easily discriminable for viewers with both typical vision and most forms of color vision deficiency. The next two colors, red and cyan, are also good for differentiation, filling in points between the first two colors along a\*. For the rest of the color scheme, we alternate between bolder and milder colors so the colors feel more balanced overall. We also considered the order of colors in our previous default to minimize the impact on existing charts that used the default colors in Plot. ## **Wrapping up: How to build a data visualization color palette that works for your organization** 1. Select primary colors that are connected to your brand or express your visual story. 2. Expand into complementary colors for differentiation in visual displays of data — and include as many colors as needed for the complexity of your data. 3. Test the color scheme in context, including for accessibility. With a considered approach to picking colors, you too can identify a color palette that enhances both the aesthetics and usability of your data visualizations. To get started with crafting data visualizations in Observable Plot, check out the [documentation](/content/plot/getting-started/index.html). \]\]>https://observablehq.com/blog/how-to-add-interactivity-observable-framework-dashboard https://observablehq.com/blog/how-to-add-interactivity-observable-framework-dashboardThu, 28 Mar 2024 00:00:00 GMTIn a previous post, we looked at how to build a dashboard for [15- and 30-year mortgage rates using JavaScript and Observable Plot.](/content/blog/how-to-build-dashboards-observable-framework-plot/index.html) Now we want to add simple interaction to the dashboard to let the user pick the year they’re analyzing. The reactive nature of Observable Framework code and Observable Plot make building interactive charts and dashboards very easy. Here’s a preview of what we’re building (you can [see it live here](/content/framework/examples/mortgage-rates/interactive/index.html), [code is available in our examples](https://github.com/observablehq/framework/blob/main/examples/mortgage-rates/docs/interactive.md)): If you haven't yet read [our previous blog post](/content/blog/how-to-build-dashboards-observable-framework-plot/index.html) or seen [the original dashboard](/content/framework/examples/mortgage-rates/index.html), please take a look at those first. We covered the dashboard in great detail there, including what the data looks like, and the different charts and how they’re built. ### How to add a year slider to your dashboard First, we want to add a way to interact with the dashboard by selecting the year. For simplicity, we’ll use a slider, or range control, here. In code, it looks like this: ``` const selectedYear = view(Inputs.range(d3.extent(pmms, d => d.date.getFullYear()), {label: 'Year:', step: 1, value: 2023})); ``` Which gives us this slider: This works by defining a constant named `selectedYear`, which is assigned the result of the [built-in `view` function](/content/framework/javascript/inputs#view(element/index.html)). This function displays the input we’re passing into it, and returns the value that the user selects. There's a lot there, so let's break down how we added interactivity to the dashboard through that line of code. First, you might be wondering about the const keyword here. The value of our constant will change when the user moves the range slider, which will re-run any parts of our code that depend on it. As long as the value of the input doesn’t change, however, this is really a constant. The input is a [range input](/content/framework/inputs/range/index.html), created by the `Inputs.range()` function from the Observable Inputs library (which is part of Framework’s standard library, so is available to use by default in the Markdown file). We need to tell it the range of values that a user can select from, which is done with this code fragment: ``` d3.extent(pmms, d => d.date.getFullYear()) ``` The `extent()` function [computes the minimum and maximum values in an array](https://d3js.org/d3-array/summarize) and returns them as a two-value array. We pass in our `pmms` array, which contains mortgage rate values for both 15- and 30-year mortgages for each month (named for the _Primary Mortgage Market Survey_). It looks like this: We’re only interested in the year of the date entry though, so we also supply an accessor function that gets the `date` value for each row and returns its four-digit year. The second argument to `Inputs.range()` is an object with a few options: ``` {label: 'Year:', step: 1, value: 2023} ``` We set the label we want to display in front of the slider, the step size (we only want whole years), and the starting value. ### Updating the tick plots for interactivity Now that we have a way of picking the year of interest, we can update our charts to include interactivity. First, let’s look at the tick charts. They were built to visualize 15- and 30-year mortgage rates over the most recent 52 weeks of data (see the [previous post for details](/content/blog/how-to-build-dashboards-observable-framework-plot/index.html)). Instead, we want to select the values for the relevant year and only plot those. To do this, we filter the `pmms` array to give us only those values: ``` const pmmsSubset = pmms.filter(d => d.date.getFullYear() === selectedYear); ``` The `filter()` function is built into JavaScript and returns only those values for which the supplied function returns true. Our anonymous function is similar to the one we used above: it accesses the entry’s `date` property, gets its four-digit year, and compares that to the year the user has selected in the range input. We can now use `pmmsSubset` instead of `pmms`. This is straightforward in the Plot code, where we replace ``` Plot.tickX(pmms, {… ``` with ``` Plot.tickX(pmmsSubset, {… ``` Similarly, the text values on the left and right of the chart depend on a value called `range`, which was computed based on the previous 53 values. ``` const range = d3.extent(pmms.slice(-52), (d) => d[key]); ``` By using our subset instead, we get the correct range for the selected year: ``` const range = d3.extent(pmmsSubset, (d) => d[key]); ``` Finally, the heavier tick mark showing the last value in the chart gets updated similarly, from ``` Plot.tickX(pmms.slice(-1), {x: key, strokeWidth: 2}), ``` to ``` Plot.tickX(pmmsSubset.slice(-1), {x: key, strokeWidth: 2}), ``` Most of this is really just data management so the right values can be passed into the calls to Plot. Every time the user moves the slider, Framework’s [reactivity](/content/framework/javascript/reactivity/index.html) automatically re-runs all the code depending on `selectedYear` (whether directly or indirectly), and the charts are recreated. In the next section, we’ll repeat this process for the line chart. ### Updating the stepped line chart for interactivity There's also a stepped line chart, which currently shows the development over the last year we have data for. We want to change it to show data for the year we selected, like this (showing mortgage rates for every week in 2013): Similar to the tick charts, the stepped line chart used the `slice()` function to extract the last 53 data values. It is drawing from the tidy version of our dataset, called `tidy`, which has separate entries for the 15- and 30-year mortgage rates for each month. It looks like this: In order to extract the 53 last values for both variables, the static version sliced off twice as many records from the end of the array: ``` tidy.slice(-53 * 2), ``` Now to get just the values for the year we’re interested in, we can do essentially the same thing as above, but for the `tidy` variable. In this case, we’re not storing the result separately since we only need it once. ``` tidy.filter(d => d.date.getFullYear() === selectedYear) ``` And voilà, we have a beautiful stepped line chart that shows just the year the user has selected! ### Updating the overall line chart for interactivity The bottom line chart showing all the data doesn’t need to update at all, since it always shows the entire range. However, we want to be able to see what year was selected. To do this, we’ll add a background rectangle that will highlight that year. ``` Plot.rectY([{year: selectedYear}], {x1: d => new Date(d.year, 0, 1), x2: d => new Date(d.year+1, 0), y1: 0, y2: d3.max(tidy, d => d.rate), fill: 'lightgray'}), ``` Let’s break down how this works. We’re adding a rectangle using the `Plot.rectY()` function. It needs data and a set of mappings. The data consists of an array with a single entry. We’re using an object for this, even though we could also just make an array of numbers (or, in this case, a single number). ``` [{year: selectedYear}] ``` We then need four coordinates to specify the rectangle, `x1`, `y1`, `x2`, and `y2`. The coordinate system is the domain of the data, so we have to specify the x coordinates using dates and the y coordinates in terms of rate percent. For `x1` and `x2`, that means creating new dates for the first day of the year we’re interested in, and the first day of the next year. The anonymous functions get the object passed to them that we just looked at above. ``` x1: d => new Date(d.year, 0, 1), x2: d => new Date(d.year+1, 0) ``` The bottom y coordinate is 0, since we’re forcing the chart to include it using the horizontal line at zero. For the top, `y2`, we want to extend to the top of the chart. We do this by using the maximum value we can find in the tidy data array. The `d3.max()` function works similarly to `d3.extent()`, which we already used above, except it only returns the maximum. ``` y1: 0, y2: d3.max(tidy, d => d.rate) ``` Finally, we specify the fill color to be slightly different from the background, so we can see the rectangle. If we make it the first entry in the marks array of the specification, it gets drawn first, so all the lines will appear on top of it. The [complete example code is available in our examples](https://github.com/observablehq/framework/tree/main/examples/mortgage-rates), or [play with the live dashboard yourself here](/content/framework/examples/mortgage-rates/interactive/index.html). ### Conclusion: Interactive dashboards enable more engagement with visualizations There is no reason a statically-generated dashboard has to be static! Using Observable Framework’s reactive nature and the flexibility of Plot, we can easily add simple interaction to a visual display of data. Interactivity lets users ask more questions and get more value out of a dashboard. Instead of picking the year, we could let the user specify an exact date range. The changes to our code would be fairly minimal. We could also make it possible to switch the different series on and off, in case somebody might not care about one or the other. This particular dataset doesn’t have many columns or levels of granularity, but it is easy to imagine being able to pick a metro area or region, for example, instead of getting values for the entire U.S. Once you start adding interactivity to charts and dashboards, the possibilities for further exploration and user engagement are endless. Learn more about building interactive data apps and dashboards with [Observable Framework](/content/framework/index.html). \]\]>https://observablehq.com/blog/data-loaders-for-the-win https://observablehq.com/blog/data-loaders-for-the-winTue, 19 Mar 2024 00:00:00 GMTNo one likes to wait. Data apps tend to be slow because they run computationally-expensive, time-consuming queries and [data analysis](/content/learn/what-is-data-analysis/index.html) on page load, making users wait. Slow data apps discourage exploration and stymie rich insights. What if data apps could be fast? With [Observable Framework](/content/framework/index.html), they can. Observable Framework’s data loaders let you precompute static snapshots of data at build time. Data loaders are programs that can be written in any language to access, wrangle, and analyze data. With data loaders, you can minimize data sent to the client and push bulky data processing “behind the scenes” to happen on build instead of on page load. That means you don’t have to choose between speed and richness: you can make fast data apps powered by any language on the back-end, with powerful interactive visualizations created in JavaScript on the front-end. Read on to learn how Observable Framework helps teams harness their existing skills to build the fastest, richest data apps — that viewers and developers can’t wait to dive into. ## But first, what’s a data loader? A data loader is a program that generates static files at build time. Data loaders can be written in any language ( _e.g._ Python, JavaScript, SQL, R, Julia, or anything else). In fact, a data loader can be literally _any_ program that: (1) runs on the computer that builds the project, and (2) is able to output information to standard output. Most often, data loaders generate snapshots of pre-processed data. For example, a data loader might: - Perform time series forecasting of web traffic in R using recent data from a secure API, then output an Apache Parquet file with both the historic and forecasted values - Access customer data from a company database, query records using SQL to only include customers who have been active on a site in the past 30 days, then emit a CSV file - In Python, read in remote sensing data from the Landsat 8 satellite, clip the data to a specific field study area, visualize the color infrared image, and generate a PNG You’ll recognize data loaders in Framework projects by their double extension, indicating the output type followed by the loader type. For example, `user-traffic.json.py` is a Python data loader that prints out a single JSON file, while `energy-demand.zip.R` is an R data loader that emits a zip archive. When you reference a data file from a markdown file ( _e.g._ with [FileAttachment](/content/framework/javascript/files/index.html)), Framework’s [routing](/content/framework/loaders#routing/index.html) will first look for that file, _then_ (if the file isn’t found) for a data loader with the matching prefix. If the data loader has been updated more recently than the cached output in `docs/.observablehq/cache`, the loader runs again and generates an updated version of the data snapshot, which gets read in by FileAttachment. Let’s consider a hypothetical example. We wrote a `.sh` data loader that accesses an organization’s user account data from a secure database, queries records to only include certain types of accounts (say, from the economy tier), then returns the individual account ID, creation date and total lifetime revenue in a single CSV. We add the data loader to `src/data/` as `economy.csv.sh`. The schematic below describes the sequence of Framework steps taken when we reference `economy.csv` in `revenue.md`: Data loader outputs don’t just integrate with Framework at build time. Local changes to data loaders are detected by the preview server and pushed to live preview, so you can see page updates while you develop your site, your data analyses and your visualizations. ## So, why use data loaders in Framework projects? ### A win for polyglot developers and teams JavaScript is the language for modern data visualization and interactive graphics on the web. But the bulk of data work isn’t creating stunning graphics and polished data apps — it’s accessing, wrangling, exploring, analyzing, and modeling data. Those tasks are most commonly done in languages like Python, SQL, or R. Data loaders bridge the language gap between data prep and data displays because they can be written in _any_ language, which means that developers and their team can extract and model data using whatever language and libraries they love. A number of data loader types, including Python, R, TypeScript, JavaScript, Julia, Rust, Go, and more are built into Framework for convenience (see the [default list of interpreters](/content/framework/config#interpreters/index.html)). Don’t see your language of choice? Don’t worry! You can use the [interpreters](/content/framework/config#interpreters/index.html) configuration option to specify any additional interpreted languages for data loaders. For example, adding the following to your project’s [configuration file](/content/framework/config/index.html) lets you write Ruby data loaders using extension `.rb` ( _e.g._`customers.json.rb`): ``` export default { interpreters: { ".rb": ["ruby"] } }; ``` You can write data loaders just like you would a normal script, with the additional requirement that the payload must be sent to [standard output](https://en.wikipedia.org/wiki/Standard_streams#Standard_output_(stdout)). It’s not always obvious how to do that in different languages, or for different output types. For example, in an R data loader you could write a CSV from a data frame stored as `myData` to standard output using [`cat`](https://www.rdocumentation.org/packages/base/versions/3.6.2/topics/cat) and [`readr::format_csv`](https://readr.tidyverse.org/reference/format_delim.html?q=format_csv#ref-usage): ``` cat(format_csv(myData)) ``` In Python (from pandas DataFrame `myData`), you could use: ``` myData.to_csv(sys.stdout) ``` And in Node (with array `myData` created in a JavaScript data loader): ``` process.stdout.write(d3.csvFormat(myData)) ``` Note: `console.log` also writes to standard output, so beware when using `console.log` ( _e.g._ for debugging) in data loaders as it can inadvertently add logged messages to your loader output. You can still debug a data loader by sending messages with `console.warn` instead, which will be shown in the preview logs. ### A win for performance If you’ve ever monologued your way through a presentation featuring a slowly-loading dashboard, you know how long seconds can feel. Users expect instant page loads. While there are [no absolute rules](https://developer.mozilla.org/en-US/docs/Web/Performance/How_long_is_too_long) for what is classified as “fast” or “slow” in web performance, whatwe _can_ say is that when it comes to page loads, **faster is better**. Data loaders can speed up dashboards and apps built with Framework in two ways. First, data loaders can perform time-consuming [data wrangling](/content/learn/what-is-data-wrangling/index.html) and analyses at build time, reducing the (sometimes slow) client-side processing needed when a viewer opens a page. In our internal web log analysis dashboard (see the [public version](/content/framework/examples/api/index.html)), a TypeScript data loader cuts page load times by performing complex database queries in SQL to generate an Apache Arrow file with records for >7 million server requests. This pushes all the heavy data processing to build-time, resulting in fast page loads — especially considering the richness of the rendered charts. You might be thinking “ _Wait, doesn’t that just mean you’ll have a longer build time?_”Yes! We like that tradeoff. We use automated, scheduled builds to keep the web log dashboard up-to-date (or, as up-to-date as we need it to be for our purposes), with near-instant page loads. The longer daily builds happen once a day, behind the scenes — not every time someone opens the dashboard on their mobile phone. Second, data loaders can be tailored to minimize the size of data sent to the client. In a data loader, you might filter data to only include recent dates, select just the variables you need for a visualization, store model outputs for selected scenarios, or aggregate raw data into summary statistics. And, you might reduce a file size in a data loader simply by converting it to a parquet file, or compressing multiple outputs as a Zip archive. Returning to the hypothetical `economy.csv.sh` data loader from the schematic above, we minimize the data by: - Filtering to only include active Economy Tier user records - Aggregating revenue values to return only the lifetime sum - Keeping just three columns: account ID, creation date, and lifetime revenue Bonus win: Data minimization is also good for data privacy and security, since only information in the generated snapshot gets shared with viewers (but “data loaders for the win-win-win- _win_” just felt like a bit much 🙂). ### A win for decision-makers It’s tempting to build data apps that pull in real-time data updated by the minute, or that allow viewers to explore changes in model outcomes at tiny increments along a range of parameter values. And with Framework, you can absolutely still do that. But do apps that allow nearly limitless, real-time data exploration help or hinder the users they’re frequently made for: business decision-makers and managers operating with limited time, and perhaps with a less robust understanding of the data and methods underlying a data visualization? We’d argue the answer is often “hinder.” As Observable founder Mike Bostock [recently wrote](https://news.ycombinator.com/item?id=39383386#:~:text=Yes!%20It%20first,to%20slow%20queries.): > It first felt counterintuitive and constraining to prepare data ahead of time, rather than just loading whatever you want on the fly. But we’ve found this to be a great discipline in practice because it forces you to think about what data you actually need to show. Most of us on the Observable Team were used to working in Observable notebooks, which we love for on-the-fly testing and tinkering with live data. While that nimble, real-time data investigation in notebooks is an essential part of [data exploration](/content/learn/what-is-data-exploration/index.html) and prototyping, it can also be a bit more haphazard than a thoughtful, “less is more” approach for building useful dashboards. Now working in Framework to create finalized and sustained data displays, we embrace a good side-effect of data loaders: they nudge us toward more carefully planned, and more strongly opinionated, data apps and dashboards. That translates to dashboards tailored for decision-makers to quickly focus on the most important scenarios, key trends, and critical comparisons, rather than sifting through an interactive data haystack. ## Are you required to use data loaders in Observable Framework projects? Nope! Data loaders aren’t mandatory in Framework projects, and sometimes aren’t even the best option. For example, you can drop CSV files (or anything else) right into your project directory for use. Or, get real time data on the client, _e.g._ via fetch or WebSocket. In fact, we use a combination of data loaders and static files in our own Framework projects and examples. Our [U.S. electric grid dashboard](https://github.com/observablehq/framework/tree/main/examples/eia/docs/data) uses a static JSON file for the U.S. base map, several (non-data loader-generated) CSVs for unchanging reference information on balancing authorities, and JavaScript data loaders to access hourly electric data from the [Energy Information Administration’s Opendata API](https://www.eia.gov/opendata/). ## Give data loaders a try! If you’re new to data loaders, we’ve got you covered: - Follow along with our [Getting started tutorial](/content/framework/getting-started/index.html) for a step-by-step introduction to Observable Framework, including how to create, modify, and use the output from a data loader to explore your local weather - Watch our [short video](https://www.youtube.com/watch?v=HtlNctuB8ho) on writing your first data loader, which follows the example from the tutorial above - Read the [Data loaders](/content/framework/loaders/index.html) section in our Framework documentation for a technical overview and examples - See how we use data loaders in our [showcase examples](https://github.com/observablehq/framework/tree/main/examples#showcase-examples), and visit our [technique examples](https://github.com/observablehq/framework/tree/main/examples#data-loaders) for reusable code to create data loaders in different languages If you aren’t yet using Observable Framework, data loaders are one great reason to try it out. Learn more about Framework at [observablehq.com](/content/site-root.html), and create your first project today! \]\]>https://observablehq.com/blog/how-to-build-dashboards-observable-framework-plot https://observablehq.com/blog/how-to-build-dashboards-observable-framework-plotTue, 05 Mar 2024 00:00:00 GMTWith Observable Framework, you can build dashboards and [data visualizations](/content/learn/what-is-data-visualization/index.html) entirely with code. But what does that look like? How does layout work, how do you prepare the data, how do you create the charts and make them interactive? In this series of blog posts, we’ll answer these questions and more. For our first post in the series, we’ll explore an example interactive dashboard that displays weekly changes in interest rates for [30- and 15-year fixed-rate mortgages in the U.S.](/content/framework/examples/mortgage-rates/index.html) from 1971 to today. It contains two different types of line charts, as well as a density chart of tick marks. We’ll look at how each of them is built using [Observable Plot](/content/blog/how-to-build-dashboards-observable-framework-plot#:~:text=is%20built%20using-,Observable%20Plot,-%2C%20and%20briefly%20touch/index.html), and briefly touch on how the layout code works. If you want to dive in directly, the [source code is available here](https://github.com/observablehq/framework/tree/main/examples/mortgage-rates). The data is loaded from an API that updates once a week. This is handled by a [data loader](/content/framework/loaders/index.html) written in TypeScript. Framework allows you to use a variety of languages to load and prepare data, which we’ll cover in more detail in a later blog post. The data loader is run as part of [Observable Framework’s](/content/framework/index.html) build process, and generates a CSV file. This file can be loaded into our dashboard much faster than accessing and querying the API on the client, allowing for almost instant page loads. ## Preparing data for interactive data visualizations Since the data is coming from a CSV file, it is naturally in a simple tabular format. In JavaScript, this translates into an array of objects, with each having three keys: `date`, `pmms30`, and `pmms15`. PMMS stands for [_primary mortgage market survey_](https://www.freddiemac.com/pmms/about-pmms.html) — the original data source — for 30- and 15-year fixed-rate mortgages, respectively. Values from early 2023 are shown below (`pmms15` values before 1990 are missing, so they end up being represented as `null` values). To create multiseries line charts in Plot (with lines for pmms30 and pmms15) more easily, we’ll transform the data into [tidy format](https://vita.had.co.nz/papers/tidy-data.pdf): ``` const tidy = pmms.flatMap(({date, pmms30, pmms15}) => [\ {date, rate: pmms30, type: "30Y FRM"},\ {date, rate: pmms15, type: "15Y FRM"}\ ]) ``` The `flatMap()` function (which is similar to `map()`) creates two objects for each entry in the initial data. Instead of the date and the two mortgage rates, we now have a `date`, a `rate`, and the `type`: “30Y FRM” or “15Y FRM”. FlatMap allows us to emit an array of multiple values from our function and flattens them all into one. The new, tidy version of the data looks like this: ## Designing dashboard layouts and using resize() The overall dashboard layout is created using CSS classes that are built into Framework. The [grid CSS classes](/content/framework/css/grid/index.html) can be used to create a grid with up to four columns that resize with the browser window. [Cards](/content/framework/css/card/index.html) can span multiple rows and columns, as you can see here (this is slightly simplified from the actual code): ```
   
[30-year card]
 
[15-year card]
 
[Stepped line chart]
``` All charts are wrapped in a helper function called `resize()`, which takes as its argument a function with one or two arguments, `width` and `height`. If only `width` is specified, the height is determined by the content that is returned by the function. In our case, this is either directly the output of a call to `Plot.plot()`, or a combination of a table and Plot. The details of the `resize()` function aren’t important to understand the rest of the code, but this is where the `width` variable comes from that you’ll see below. ## How to build a line chart We’ll start with the two line charts. While they look different, their definitions are actually very similar. First, let’s look at the last chart on the dashboard, since it is the most traditional chart here: Here is the [Plot](/content/plot/index.html) definition for the line chart above: ``` Plot.plot({ width, color, y: {grid: true, label: "rate (%)"}, marks: [\ Plot.ruleY([0]),\ Plot.lineY(tidy, {x: "date", y: "rate", stroke: "type", tip: true})\ ] }) ``` First, we have the `width` variable to set the chart’s size as well as a `color` constant, which keeps the color mappings of the 30- and 15-year data series consistent across charts (just `width` turns into `width: width`, a little trick to save some typing when your variables already have the obvious names). Next, we define a vertical axis using the `y` property, to turn on the grid and add a label at the top of the axis. Then there are two marks defined here. The first one uses [Plot’s ruleY mark](/content/plot/marks/rule/index.html), which defines a horizontal line (similar to the tick mark we use below, but when its length isn’t specified it fills the entire width of the chart) to add a horizontal line at y=0 to make the chart look nicer and force Plot to include 0 on the vertical axis. The second mark draws the actual line of mortgage rate values. Here it is again: ``` Plot.lineY(tidy, {x: "date", y: "rate", stroke: "type", tip: true}) ``` It uses another line-related mark, [lineY](/content/plot/marks/line/index.html), which is used to draw line charts. Both line charts use the `tidy` variable we created earlier, where each row or object has values for the `date`, `rate`, and `type` (30- or 15-year mortgage). These are mapped directly to mark properties here, with the `date` going to x, `rate` to y, and the `type` to stroke. Since there are two types, Plot creates two lines and assigns colors to them based on the `color` property we defined earlier. Finally, the `tip` property turns on the default tooltip. Typical line charts like this one draw lines between the points under the assumption that values change continuously. That isn’t true for mortgage data, which only contains one value for each week and we can’t assume that they change linearly over time (like temperatures would). However, since we’re looking at over 50 years and around 2,700 values in this chart, there are enough data points so stepped and regular line charts look virtually the same. For the top line chart that just shows 52 weeks, however, it is important to make it clear that these are the only values we have in that period. We can do that by creating a stepped line chart that draws horizontal lines between the points in time when the data changes, and then jumps vertically to the next value. In Plot, this is a simple option. The only thing that changes from the other line chart is the mark definition: ``` Plot.lineY(tidy.slice(-53 * 2), { x: "date", y: "rate", stroke: "type", curve: "step", tip: true, markerEnd: true }) ``` There are two main differences here. First, we only take the most recent values from the `tidy` variable. Since we have two data series for the two mortgage durations, and we need 53 data points to draw 52 lines between them, that ends up being 53⨉2 values. The other key difference is the additional option, `curve: "step"`. This creates the stepped line chart. One other addition is the `markerEnd` option, which draws a dot at the end of each line to indicate that these are the most recent values we have. ## How to build tick charts The charts at the bottom of the two smaller cards show the most recent 52 mortgage rate values for each series (30- and 15-year fixed rates) as ticks. This gives a sense of the distribution of the values and if they were they higher or lower on average over this time. The most recent value is shown as a heavier, longer black line. Like the line charts, the tick charts are created entirely in Observable Plot, including the labels for the range of values on either side. The content of the card is created inside a function that can be called for either the 30- or 15-year mortgage durations. It returns the big number and table at the top of the card, as well as the chart. We’ll focus only on the chart here, but feel free to explore how the other parts are created in a reusable JavaScript component. Here’s the code for the tick charts. No worries, we’ll break it down in a moment: ``` Plot.plot({ width, height: 40, axis: null, x: {inset: 40}, marks: [\ Plot.tickX(pmms.slice(-52), {\ x: key,\ stroke,\ insetTop: 10,\ insetBottom: 10,\ title: (d) => `${d.date?.toLocaleDateString("en-us")}: ${d[key]}%`,\ tip: {anchor: "bottom"}\ }),\ Plot.tickX(pmms.slice(-1), {x: key, strokeWidth: 2}),\ Plot.text([`${range[0]}%`], {frameAnchor: "left"}),\ Plot.text([`${range[1]}%`], {frameAnchor: "right"})\ ] }) ``` Let’s first look at the beginning of the code for the chart, which sets up its basic parameters. ``` Plot.plot({ width, height: 40, axis: null, x: {inset: 40}, ``` It defines the size of the chart using the `width` and `height` fields. The `width` variable comes from the `resize()` function mentioned above. Since we don’t need an axis, we set that to null, and define a horizontal inset of 40 pixels to make some room for the text labels on both sides. The definition of the marks begins with [Plot’s tickX mark](/content/plot/marks/tick/index.html), which simply adds vertical ticks along a range of values: ``` marks: [\ Plot.tickX(pmms.slice(-52), {\ ```\ \ We provide its data in the form of the `pmms.slice(-52)`expression, which gives us the last 52 values of the `pmms` array. The rates are updated every week, which means we get about one year’s worth of data back from the most recent data value.\ \ Following this, we define the mark’s properties:\ \ ```\ x: key,\ stroke,\ insetTop: 10,\ insetBottom: 10,\ title: (d) => `${d.date?.toLocaleDateString("en-us")}: ${d[key]}%`,\ tip: {anchor: "bottom"}\ })\ ```\ \ Going from the bottom to the top, we…\ \ - set the `tip` and `title` properties to get a tooltip when we hover over the lines,\ \ - use `insetBottom` and `insetTop` to shorten the lines vertically so we can later add a longer line as an overlay,\ \ - use the `stroke` value is defined earlier in the function to select a color that matches the global 15-year or 30-year data series colors used on all the other charts.\ \ \ Finally, the `x` property is set to a variable `key`, which is created earlier in the function:\ \ ```\ const key = `pmms${y}`;\ ```\ \ The y value is passed into the function as a parameter and is either 15 or 30 for the respective duration of the mortgage. Appending that number to the `pmms` string gives is the name of the field in the `pmms` data array (e.g. “pmms15”), which Plot uses to access the values.\ \ That wraps up the short ticks that make up the majority of the chart. Next, we add the heavier tick for the most recent value:\ \ ```\ Plot.tickX(pmms.slice(-1), {x: key, strokeWidth: 2}),\ ```\ \ The `slice()` function here gives us only the last entry in the array. The definition for `x` is the same as above, and then we only set the `strokeWidth` to get a heavier line. Since there’s no inset here, the line spans the entire height of the chart, making it more prominent. Marks are rendered in the order in which they are defined in Plot, so this line gets drawn on top of the other ones.\ \ Finally, we add the two text labels on either side of the ticks using [Plot’s text mark:](/content/plot/marks/text/index.html)\ \ ```\ Plot.text([`${range[0]}%`], {frameAnchor: "left"}),\ Plot.text([`${range[1]}%`], {frameAnchor: "right"})\ ```\ \ The square brackets above create arrays with a single value each as the dataset for Plot. The `range` variable is created earlier in the function using `d3.extent()` to get the extreme values of the data range. The `frameAnchor` property pushes the labels to the left or right, respectively.\ \ And voilà, we have created a tick chart!\ \ ## Wrapping up: Creating expressive, interactive charts and dashboards with code\ \ Creating dashboards and data visualizations with code, especially using Observable Framework and Observable Plot, gives us a lot of freedom and power. We can create any chart we want, combine different kinds of charts, and build visual displays of data that we might not even think of as charts, like the tick charts. While you can build very complex data visualizations with Observable Plot, they are built from simple definitions and can be broken down into their individual data mappings.\ \ This example uses Plot and doesn’t include interaction (except for the tooltips). There are many ways to make [charts interactive](/content/blog/five-techniques-to-improve-chart-annotations/index.html) and create different types of visuals using D3, etc. Learn more about adding interactivity to data visualizations and dashboards by including [chart annotations](/content/blog/five-techniques-to-improve-chart-annotations/index.html).\ \ \]\]>https://observablehq.com/blog/observable-2-0 https://observablehq.com/blog/observable-2-0Thu, 15 Feb 2024 00:00:00 GMTToday we’re launching [Observable 2.0](/content/product/index.html) with a bold new vision: an open-source static site generator for building fast, beautiful data apps, dashboards, and reports. Our mission is to help teams communicate more effectively with data. Effective presentation of data is critical for deep insight, nuanced understanding, and informed decisions. Observable notebooks are great for ephemeral, _ad hoc_ data exploration. But notebooks aren’t well-suited for polished dashboards and apps. Enter [Observable Framework](/content/framework/index.html). With Framework, you can build the best data apps your team has ever seen. Framework combines the power of **JavaScript on the front-end** for interactive graphics, with **any language on the back-end** for data preparation and analysis. SQL, Python, R, Rust, Go… you name it. It’s the polyglot programmer’s dream. Everything you need is at your fingertips: interactive charts and inputs, responsive grids, themes, dark mode, keyboard-friendly navigation, and more. And because it’s code, there’s no limit to customization! Framework is free and [open-source](https://github.com/observablehq/framework). Projects are just local files. Use your favorite editor, preview locally, check it all into git, write unit tests, add CI/CD, even work offline. You can host projects anywhere or deploy instantly to Observable to share them securely with your team. Observable Framework solves the “last mile” problem of data apps: loading data. Conventional dashboards are slow because they run queries on view while the user waits; Framework’s data loaders run _on build_ so that pages load instantly. And because data loaders run on your servers, you control privacy and security. If you’re ready to dive in, visit our [Getting started tutorial](/content/framework/getting-started/index.html), or open a terminal and run: npm init @observablehq If you’d like to hear more about why we built Framework, please read on. ### Beyond notebooks **📓** This moment — Observable 2.0 — reflects lessons learned over many years. We believe the lightweight, collaborative nature of computational notebooks makes them ideal for exploring data and answering _ad hoc_ questions. We founded Observable in 2016, pioneering a [reactive, web-first approach](https://medium.com/@mbostock/a-better-way-to-code-2b1d2876a3a0) to notebooks and seeking to make data visualization easier, more practicable, and more social. We dreamed that notebooks might be the “one ring to rule them all” — powering not just notes, but apps, dashboards, and reports. Yet no single interface can excel at every task. As cool as reactive notebooks are, a notebook can’t compete with a custom web app in terms of user experience. Notebooks are constrained by: - A single-column, narrow layout - Low visual information density - Always-visible editor chrome These same limitations make notebooks great for tinkering and learning — the code is always at your fingertips, adjacent to the output — but not so great for presentation. To fill that latter role, we need better data apps. A good data app embodies an empirical perspective; it fosters a shared understanding. Whereas notebooks tend to be for individuals, data apps are more often for a team. And whereas notebooks tend to be transient byproducts of point-in-time exploration, data apps often sustain value over time as people return to see how things change. The differences between notebooks and data apps extend to development. A notebook editor desires speed: jotting down thoughts, running a query, sketching a chart. A data app developer prioritizes correctness, performance, and maintainability: making careful, deliberate changes that others depend on, favoring code review and testing before publishing. We had three goals in mind when we set out to reimagine data app development: 1. **A better developer workflow** — meeting the needs of developers 2. **A better user experience** — the “proof is in the pudding” 3. **A better data architecture** — solving the “last mile” problem ### A better developer workflow **👩‍💻** Modern development is built on [files](https://stephango.com/file-over-app). Files have myriad strengths, but the strongest is interoperability. When every tool uses files, it’s far easier to incorporate a new tool — and now Observable — into your workflow. This isn’t just about using your preferred text editor. Now you can bring your own source control and code review system, too. You can write unit tests and run linters. You can automate builds with continuous integration or deployment. You can work offline. You can self-host. You can generate or edit content programmatically, say to format code or to find-and-replace across files. As we break new ground with Observable Framework, we’re further improving interoperability by adopting vanilla JavaScript syntax. And we’re deprecating `require` in favor of modern ES `import`. These changes make Observable easier to learn, and to share code with other applications. (We’ll port these improvements back to Observable notebooks in the future.) Check out our [examples on GitHub](https://github.com/observablehq/framework/tree/main/examples). ### **A better user experience 😍** A toolmaker can’t care only about the developer experience — what does the developer experience matter if the resulting app is not demonstrably better? The merit of a creative tool should be judged by the quality of its creations, not its process. Or: “the proof of the pudding is in the eating.” We believe that well-designed tools help developers build more efficiently by focusing their efforts on high-value work. We favor opinionated tools, with defaults and conveniences that foster a good user experience. We nudge you into [the pit of success](https://blog.codinghorror.com/falling-into-the-pit-of-success/). Framework’s lightweight Markdown syntax — with light and dark mode, thoughtful colors, responsive grids, and built-in navigation — gives you beautiful pages from the start. It’s highly customizable if you need it, but it’s quick to get started with batteries included. Most importantly, Framework’s data architecture practically forces your app to be _fast_ because data is precomputed. Performance is critical for dashboards: users don’t like to wait, and dashboards only create value if users look at them. Slow dashboards waste time. (And you certainly don’t want your database and dashboard falling over under load!) ### A better data architecture Every data visualization requires data. Obviously. But less obviously, each data visualization requires highly-specific data prepared with that visualization in mind. In fact, most of the work of visualization isn’t choosing visual encodings or laying out axes or visualizing _per se_ — it’s preparing data. As I wrote [previously](/content/@mbostock/10-years-of-open-source-visualization/index.html), > working with data should be 80% of the work of visualization. Visualization is the end result of analysis — the visible manifestation of data, to be seen, shared, and appreciated by experts and non-experts alike — and as such it sometimes gets too much credit. To produce a visualization, one must first find data, clean it, transform, join, model, etc. Working with data is sometimes needlessly denigrated as “janitorial” when it represents the critical step of understanding the data as it is, warts and all. Given how much work goes into preparing data, it follows that developers want - to use any language (say Python or R or SQL), - to use any library (say NumPy or dplyr), - to use any data source (database, data warehouse, API, files, _etc_.), and - to crunch data ahead of time (offline) while still leveraging JavaScript in the browser for interactive graphics. Framework’s data loaders solve this “last mile” problem by computing static data snapshots at build time. These snapshots can be highly-optimized (and aggregated and anonymized), minimizing the data you send to the client. And since a data loader is just a fancy way of generating a file on-demand (with clever caching and routing), loaders can be written in any language and use any library. This flexibility is not unlike [CGI](https://en.wikipedia.org/wiki/Common_Gateway_Interface) from 30 years ago, and [Unix pipes](https://en.wikipedia.org/wiki/Pipeline_(Unix)). And since data loaders run on your servers, viewers don’t need direct access to the underlying data sources, and your dashboards are more secure and robust. The speed of modern data warehouses is astonishing. But far too often something is missing for new analysis — some untapped data source, some not-yet-materialized view for a query to run at interactive speeds. Framework’s data loaders let you bypass these hurdles and produce a fast dashboard without “heavy lifting” in your data warehouse. And once your analysis demonstrates value, you can shift work to your data warehouse and simplify your data loaders. Framework lets you build faster and quickly validate your ideas. We believe Framework will change how you think about data, and effect a better user experience. And by securely hosting apps alongside notebooks, Observable now offers an end-to-end solution for data analysis and presentation. ### Thank you **🙏** We wouldn’t be here without the support, feedback, and encouragement from you — our community. Thank you for using Observable notebooks, Observable Plot, and D3. We’re thrilled to share Observable Framework with you now, and can’t wait to hear what you think. To learn more about Framework, [read the docs](/content/framework/index.html). To share your questions or feedback, please [visit our forum](https://talk.observablehq.com/latest). \]\]>https://observablehq.com/blog/scheduled-notifications https://observablehq.com/blog/scheduled-notificationsTue, 14 Nov 2023 00:00:00 GMTStay in the loop and never miss a beat Available now to Pro and Enterprise workspaces, Observable’s scheduled notifications are a set of beta features that allow you to share charts, distribute query results, and send reports programmatically. Developers can now enjoy a more efficient workflow with automated updates that ensure timely decision-making without the need for manual report distribution or frequent data refreshes. Keep everyone in the loop with scheduled email or Slack updates so your team can make informed decisions when it matters most. ## Notifications at your fingertips You can use our [Notifications](/content/documentation/notebooks/notifications/index.html) feature by invoking `NotificationClient` in a cell. This creates a message payload that, when paired with our [Schedules](/content/documentation/notebooks/schedules/index.html) feature, will dispatch a range of content outputs: - Personalized messages (string) - PNG images (automatically convert any SVG or canvas element) - CSV - JSON - File blob (attached as a file) These can be sent directly via email or Slack — kick-starting real-time discussion on topics like sudden data spikes or rounding up the weekly analytics report. ``` notify = (await NotificationClient("email:you@example.com")).send( `Observable Plot downloads yesterday: ${downloadsYesterday} (${percentile})`, chart ) ``` Before `NotificationClient` can dispatch messages, you must set the notebook to run at specific times using the Schedules feature. This can be done in just a few clicks and it will ensure consistent, automated notifications. ## Automation and communication made easy Set your notebooks to run on autopilot with [Schedules](/content/documentation/notebooks/schedules/index.html). Found in the notebook sidebar, the Schedules feature enables you to automatically run notebooks at set intervals. The default timing options include every 10 minutes, hourly, or every 24 hours. Schedules is the feature that powers Notifications, but it can also be used to perform any other function that an Observable notebook cell can already do in a browser client session. This includes querying databases or accessing APIs. Looking for a more tailored schedule? You can add logic so that your scheduled notebook only runs at specific times or under certain conditions. _Example: Only run if it’s a Monday._ ``` { if (new Date().getDay() === 1 /* Monday */) { return slack.send("Happy Monday!") } } ``` ## Safeguarding your data throughout automation Observable has doubled down on [data security and privacy](/content/documentation/security/data-security-and-privacy/index.html) in regards to these features. While Observable notebooks traditionally operate on the client side, ensuring that sensitive data does not leave your browser without your control, we’ve taken additional steps to enhance the level of security to our server-side processes as well. When it comes to automated features, such as scheduled notebook runs, Observable executes your notebook code on our secure servers. This makes it possible for your notebooks to run smoothly from anywhere. Our logging is minimal and strictly for the reliability of scheduled runs or for debugging purposes, ensuring the security of your data. During these operations, we function as a secure conduit: while we execute the queries necessary for your notebook's functionality, we do not and cannot retain the results. Instead, they are directly passed through to the downstream notification service of your choosing, maintaining the confidentiality and integrity of your data throughout the process. For more detailed information on how we do this, see [Scheduled notebooks and your data](/content/documentation/notebooks/schedules#data-policy/index.html). ## Empowering teams with proactive tools Our Notifications feature sprang from the authentic requirements of our own developers, testament to Observable’s commitment to solving real problems that data developers routinely face. > “As a professional developer who has used Observable since 2018, scheduled notifications are, for me, the most important expansion in fundamental platform capabilities since we added database connectors in late 2019.” > — Toph Tucker, Software Engineer @ Observable Notifications has increased engagement across our own internal Slack channels, which have become hubs of timely insights, from discussions surrounding new org sign-ups to user engagement details. This isn't just about sending notifications; it's about enhancing the situational awareness across your organization. We’re eager to see how these new capabilities will improve your workflows and your team’s collaboration. Watch our blog for more updates from Observable and happy coding! * * * _Links featured in this post:_ - [_Schedules documentation_](/content/documentation/notebooks/schedules/index.html) - [_Notifications documentation_](/content/documentation/notebooks/notifications/index.html) - [_Data security and privacy_](/content/documentation/security/data-security-and-privacy/index.html) \]\]>https://observablehq.com/blog/improving-ai-assist https://observablehq.com/blog/improving-ai-assistMon, 02 Oct 2023 00:00:00 GMTAI Assist just got a big upgrade with improved support for Observable Plot and D3. Plus, it’s now even easier for you to take advantage of this feature without having to use your own OpenAI API key. Let’s explore what’s new with AI Assist and how you can get started using it today. ## **Use AI Assist to write Observable Plot or D3 code** We created AI Assist to help users work with data more efficiently in Observable. Initially, this beta feature helped users write SQL and functioned as a pairing partner that could answer general questions about data visualizations or error codes. We’ve continued to invest in expanding the functionality of AI Assist and are excited to announce improved support for Observable Plot and D3. AI Assist can now be a reliable pairing partner throughout more of your data workflow. Our new functionality reduces the burden of experimentation by building charts in Observable Plot or D3 for you. This allows you to more efficiently explore your data visually and can help you generate useful starting points to customize based on your specific goals. For users who are less familiar with Observable Plot or D3, AI Assist can also help you modify existing charts. ## **Explore Observable and get answers with our experimental documentation agent** AI Assist can also be used as a documentation agent as the model has been tailored to return results based on Observable’s documentation pages. Ask the chat questions about features, functionality, or troubleshooting without leaving your current workflow in the notebook to get answers. ## **Try AI Assist on us** For the month of October, all new and existing Pro and Enterprise users can take advantage of 70,000 tokens to use AI Assist for free. This means that you no longer need to have an OpenAI API key to try out this feature. For current workspaces, the quota is available now. If you aren’t yet using Observable, simply [sign up for a free Pro trial](/content/signup/index.html) and you’ll automatically have a quota of tokens to use during your trial period. You can read more about these developments and experimental features in our [release notes](/content/@observablehq/release-notes/index.html). ## **FAQ** ### **Is AI Assist available to free users?** AI Assist is still available only for Pro or Enterprise workspaces, but any workspace that upgrades from Free during October 2023 is eligible for this promotion. Please be aware that accounts with a 100% off coupon applied (free usage) are not eligible for this promotion. ### **How are the tokens applied and who can use them?** 70k tokens are issued _for each workspace seat_ and are made available across the entire workspace. For example, if you have a total of two workspace seats, then you will be issued 140k tokens that can be used by all users within that workspace. ### **What happens if you use all the tokens in your workspace?** Once you’ve used up your token quota, you’ll get a notification prompt to enter in your OpenAPI key. ### **Can I buy / add more tokens?** The promotional token issuance for AI Assist is a one-time offer for the month of October and cannot be purchased or added to. To continue to use the AI Assist feature beyond the promotional tokens, users are required to establish and link their OpenAI API account within their Observable workspace. ### **Are the tokens for everyone on the team or just specific seat types, such as the account owner?** Tokens are available to everyone on the team. \]\]>https://observablehq.com/blog/september-release-recap https://observablehq.com/blog/september-release-recapThu, 28 Sep 2023 00:00:00 GMTDoing data work is hard enough as it is — you don’t need your tech stack to add to the complexity. In addition to streamlining your data workflow, we are improving the experience of using Observable so you can stay focused and keep moving your business forward with data. This month’s releases will help you do just that. **Better workspace navigation** You’ll notice that we’ve changed the [workspace navigation](/content/@observablehq/release-notes#cell-3084/index.html) to help you focus on the content, settings, and people that you work with in Observable. We’ve modified the top navigation to remove and relocate links that distracted users from the workspace context. The side navigation was also re-worked so you can more easily focus on data work and team management. In addition to the navigation refresh, we’ve also provided the ability for users who work in multiple workspaces to select a default workspace that will remain consistent as they move through the platform. Less toggling between workspaces means more productivity. **Download as PNG** The new navigation updates make it easier to dive back into your work. Once you’re in a notebook, there’s also a new feature to discover in the cell menu — Download as PNG. Rather than taking screenshots, you can download graphical cells as PNG files. The magic of this feature lies in the fact that the image is captured locally, by your browser, which means you can zoom on a Leaflet map, interact with a data visualization, let a WebGL animation run its course, and take a picture of its current state. You can learn more about this improvement in the [release notes](/content/@observablehq/release-notes#cell-3082/index.html). We hope these new releases help you maintain your data flow and keep uncovering insights and moving your work forward. To keep tabs on updates on an ongoing basis, be sure to check out our release notes. As always, we want to hear from you, so please share your thoughts or suggestions in our [feedback repo](https://github.com/observablehq/feedback). \]\]>https://observablehq.com/blog/observable-and-d3-visualizations https://observablehq.com/blog/observable-and-d3-visualizationsWed, 20 Sep 2023 00:00:00 GMTThe D3 community is an inspiration to all of us at Observable. It’s been 12 years since our co-founder, [Mike Bostock](/content/@mbostock/index.html), created [D3](https://d3js.org/), the JavaScript library loved by many for its ability to produce dynamic, interactive, and stunning works of data visualization. So while D3 (or [D3.js](https://en.wikipedia.org/wiki/D3.js)) is a staple among practitioners, it’s not always known how closely connected it is with Observable’s platform. In fact, Mike built Observable with D3 in mind. Our platform supports the creation of visualizations with D3, offering developers and data teams a robust ecosystem to create expressive, interactive [data visualizations](/content/blog/observable-and-d3-visualizations/index.html). By bringing developers and data teams together, the Observable platform streamlines collaboration so they can spend more time creating expressive data visualizations, and automate basic data prep and data app maintenance. By building on Observable, developers can work more quickly with useful starting points, such as reusable [D3 examples](/content/@d3/gallery/index.html) from a large community of D3 practitioners. In the below post, we walk through Observable’s commitment to supporting D3, our mission to build a platform for developing and hosting interactive data visualizations with the library. ## Our commitment to D3 Observable’s aim is to make developers’ lives easier with product features and functionality that accelerate insights and exploration of what can be done with data. In addition to maintaining and supporting D3, Observable also actively maintains Observable Plot, a concise open-source JavaScript library for expressive data visualization, and Observable Framework, our open-source static-site generator for data apps and dashboards. Here’s how Observable supports the D3 community: - **We want to show you how to use D3 on Observable**. When you build with D3 in Observable, you can instantly connect to your data, code faster than ever, and accelerate your team’s analysis. - **We want to make it easier to learn D3.** Observable’s coursework is intended to support the developer community to share and learn from each other. - **We recognize we’re not the only source for D3 use.** And we’re OK with that.The D3 community uses its tools for building, and we recognize and appreciate those preferences. If anything, we want the D3 community to see what’s possible on Observable, so you can decide on what’s best for your needs. [iframe](https://www.youtube.com/embed/eTaR400-lnk?si=rkSxdN6KROe0kdi1) ## Reflecting on D3's origin and impact When D3 celebrated its 10th anniversary, Mike put down some [lasting thoughts](/content/@mbostock/10-years-of-open-source-visualization/index.html) that highlight the best ways to get value from D3. The “lessons learned,” as he called them, helped community members take inventory of what’s possible and how to maximize their time building. One of the most striking observations a few years later is how he articulated the cost of being too enamored with technical “whizbangery,” a nicely worded description from the creator on the visual impact that can be achieved through D3. Mike always felt that interaction and animation had a huge impact on what was built with D3, especially for audiences who were familiar with the output. But Mike was quick to point out that the true spark for the building was the findings from the effort. [iframe](https://www.youtube.com/embed/Sqeymt1mqAw?si=GktBFq4Fv5K48ihe) “Knowing this, you may be tempted to add these features to visualizations without fully appreciating the downsides, such as added complexity and hiding valuable information behind interactive controls,” he wrote. “And worse, because these features are often challenging to implement, they may distract from the far more important yet ‘boring’ task of finding and communicating insight!” “This is not a moral judgment,” he added. “I’m not saying you’re bad for doing this. I’m guilty of this myself. But the pitfall is real. Focus on the static form first and foremost. This may be the only thing some readers see. Don’t allow technical matters ( _i.e._, web development) to eat too much of your attention. And don’t fear plain visualizations: it’s the insight that matters, not whether it’s gussied up.” So if plain visualizations are not to be feared, let’s continue to explore the possibilities of D3, with or without Observable, being the platform that enables [data exploration](/content/learn/what-is-data-exploration/index.html) and data visualization. Observable remains committed to the D3 community. To get started with building data visualizations with Observable, [sign up today](/content/signup/index.html). \]\]>https://observablehq.com/blog/august-release-recap https://observablehq.com/blog/august-release-recapTue, 29 Aug 2023 00:00:00 GMT This month we shipped two new releases to help developers work more quickly in Observable. These updates — AI Assist and the Add Cell Bar — bring greater efficiency to your data work so you can move past toil and spend more time on higher level tasks. See how these updates help you maximize your time in ways both big and small. ### AI Assist Accelerate your data visualization work with AI Assist. Available now as a beta feature for Pro and Enterprise workspaces, AI Assist can help you more quickly write code and pair with you in your data visualization work. There are two main ways you can use AI Assist right now to unlock endless possibilities. First, you can rely on AI Assist to write SQL queries, debug error codes, and optimize your queries. This feature can even craft complex, multi-table queries. When you use our AI chatbot in this manner, you can up-level your skills while saving time for more essential tasks. Additionally, you can pair with AI Assist throughout your [data analysis](/content/learn/what-is-data-analysis/index.html) workflow. It can even make recommendations about chart types, provide suggestions, and help you get unstuck. Outside of these two core use cases, AI Assist also offers experimental features you can try, such as providing assistance with [Observable JavaScript](/content/@observablehq/observable-javascript/index.html), generating basic charts in [Observable Plot](/content/plot/index.html), or building more complex charts in [D3](https://d3js.org/). Currently, to get started using AI Assist you’ll need to provide an OpenAI API token. [Learn more about AI Assist in our documentation](/content/@observablehq/introduction-to-ai-assist/index.html). ### Notebook improvements Our newest updates to the notebook include the creation of an Add Cell Bar and a refresh of the Add Cell Menu. These changes simplify everyday workflows, such as inserting cell types, by bringing common building blocks to the forefront. The new Add Cell Bar makes it easy to find and insert basic cell types in the notebook. In the Add Cell Menu, we’ve made it easier to search for components, build from sample data, or connect to your data sources. Give these new improvements a try by signing into your Observable account or check out our [release notes](/content/@observablehq/release-notes#cell-3067/index.html) for more information. ## Conclusion We hope these new releases help you move past blockers so you can more quickly build the data visualization, dashboards, and data apps that will push your business forward. As always, we want to hear from you. Let us know what you think about these new updates on our [feedback repo](https://github.com/observablehq/feedback). \]\]>https://observablehq.com/blog/ai-assist https://observablehq.com/blog/ai-assistTue, 15 Aug 2023 00:00:00 GMTImagine always having someone to pair with on data work — to brainstorm with when you get stuck, to tap on the shoulder when you don’t understand an error, to help write code. Now, with [AI Assist](/content/@observablehq/release-notes#cell-3063/index.html) in Observable, you do. Ask questions and get help right in your notebooks. Available in beta for [Pro and Enterprise users](/content/pricing/index.html), this feature taps into recent advances in generative AI to help you fast-forward past tedious tasks and accelerate your data workflow. ## **What you can accomplish with AI Assist** With AI Assist, there’s now no need to write boilerplate SQL code, allowing you to put your energy toward higher level tasks. You can rely on AI Assist to write SQL queries, including complex, multi-table queries by simply granting access to your schema and describing the desired results. Or, perhaps you’ve written a query that you want to edit or improve. Ask AI Assist to suggest optimizations as you continue to refine, iterate, and wrangle your data. AI Assist is built on GPT-4, so in addition to writing code, AI Assist can function as a pairing partner if you get stuck in your [data analysis](/content/learn/what-is-data-analysis/index.html) workflow or need a recommendation for how to approach a problem. Spend less time debugging by using AI Assist to explain how a query works or how to correct errors. Discover different ways of presenting data by asking for recommendations on which type of chart to use. You can also give our experimental features a try by asking AI Assist to write code in [Observable JavaScript](/content/@observablehq/observable-javascript/index.html), [Observable Plot](/content/plot/index.html), or [D3](https://d3js.org/). ## **How to use AI Assist** AI Assist is available in the right navigation menu in the notebook, so you don’t have to toggle between different tabs or tools. AI Assist is built on GPT-4 and currently requires that you bring your OpenAI tokens to run queries. This means that the first time you use AI Assist you’ll be prompted to enter your existing OpenAI API key. Once you do so, you’ll be able to ask AI Assist questions and converse with it to fine-tune the response. If you ask AI Assist to write code for you, you can choose to allow AI Assist to overwrite the code in your notebook cell or copy and paste the code yourself. If you'd like to learn more about how AI Assist works, go deeper into how to use this feature [in our documentation](/content/@observablehq/introduction-to-ai-assist/index.html). ## **What’s next for AI Assist** As you use this beta version of AI Assist, please let us know what you think by providing feedback on our [feedback repo](https://github.com/observablehq/feedback). AI Assist is currently in beta and we are excited to continue investing in features that use AI to improve your [data exploration](/content/learn/what-is-data-exploration/index.html) and analysis workflow. You can read more about these developments and the current experimental features in our [release notes](/content/@observablehq/release-notes#cell-3063/index.html). \]\]>https://observablehq.com/blog/fall-23-essential-features https://observablehq.com/blog/fall-23-essential-featuresThu, 27 Jul 2023 00:00:00 GMTIf you’re teaching with Observable, you’re likely thinking – or trying not to think about – your upcoming classes. Our engineers have been busy over the past few months releasing great features that we want to spotlight while you’re planning your next semester of instruction. Make sure you [contact us](/content/@educators-network/contact-edu/index.html) about our education discount! ## **Plot 0.6.2** We’ve launched an update to Observable Plot, including [tree marks](/content/plot/marks/tree/index.html), [tip marks](/content/plot/marks/tip/index.html), and [plot.auto](/content/@observablehq/plot-auto/index.html), a new feature that automatically chooses the best mark type based on your data. We’ve also launched a great new [resources site](/content/plot/index.html). You can use Plot to get students familiar with the concepts of data viz and Observable before they begin coding. They can learn by engaging with inputs, editing code in tinker mode, or converting Plot to Javascript directly inside the notebook. [iframe](/content/embed/@observablehq/plot-gallery@latest?cells=geo/index.html) _Learn more about the_ [_Geo mark_](/content/plot/marks/geo/index.html) _, a recent feature we've added to Observable Plot._ ## Chart Cell The [chart cell](/content/blog/chart-cell/index.html) is a brand new cell type that uses Observable Plot to intelligently create visualizations. You just need a data source and at least one attribute to start exploring data visually without writing a line of code. The chart cell acts as a springboard for more detailed instruction. You can easily teach high-level concepts using Observable Plot and then convert the cell to editable code for further [data exploration](/content/learn/what-is-data-exploration/index.html). _Chart cell makes it possible for learners of all experience levels to quickly create data visualizations._ ## Data Table Cell Here’s another new cell type to enjoy! Create a [data table cell](/content/@observablehq/data-table-cell/index.html) from a file attachment or JavaScript array by creating a new cell and selecting "Data table" from the cell menu. Simply click on a cell in the table for a menu of useful filters. You can explore and clean data right in a data table cell by clicking the three dots menu at the top of a cell and adding a derived column of data using a code editor right inside the cell. ## Explore more data We recently launched support for [DuckDB](/content/@observablehq/duckdb/index.html), so you can query any tabular data with SQL. DuckDB supports a variety of popular file formats, including CSV, TSV, JSON, Apache Arrow, and Apache Parquet. Our DuckDB database client implementation is based on previous work by the CMU Data Interaction Group. [Visit our site](/content/data-integrations/index.html) to see the increasing number of ways you can connect to data with Observable. ## Forking forward We can’t say enough about how useful forking is for teaching with Observable. Do you use forking in class? > “Beforehand, I’ll create two notebooks: a skeleton version and a fully complete version with the answers. During class, we work through the skeleton version together. After the lectures, I will publish the full version.” Jon Froehlich, Associate Professor in Computer Science at the University of Washington [This powerful feature](/content/@observablehq/fork-suggest-merge/index.html) to share and maintain teaching content, easily distribute assignments, and provide low-risk learning opportunities for students. They can fork worked examples and deconstruct or build on them for class projects. ## Connect and share [Tags](/content/@observablehq/tags/index.html) can help you and your students organize your classroom and discover new examples. When you click a tag, Observable searches all the notebooks in the community. You can use unique or course-specific tags to group notebooks within a team or classroom (ex: dv344-sec1, final-pt2, and team starbug), or more general tags that surface concepts across the Observable community (ex: stats, inputs, and plot), giving students resources beyond the classroom. [Follow](/content/@observablehq/following/index.html) users and teams to see their notebooks in the feed on your Observable homepage. Observable creators can provide inspiration and support for your classroom. You can also use this feature to check out skills grads will need to be competitive, stay current with the work of industry experts, and engage real-world examples. [Guests](/content/@observablehq/team-creation-and-administration?collection=@observablehq/accounts-and-workspaces#cell-127/index.html) can be added to your workspace with restricted access to notebooks, secrets, and databases. You can add colleagues from your institution, project stakeholders, and others who may need limited access to your workspace without giving up control of materials. You’ll find plenty of ways to engage students and manage classwork with Observable. As you’re planning your semester, consider how you might incorporate these useful features into your upcoming semester. Adding them can have a big impact! You can always learn more about recent features [on our blog](/content/blog/index.html), review the latest updates in our [release notes](/content/@observablehq/release-notes/index.html), and [subscribe to our newsletter](/content/@observablehq/plot-twist-newsletter-signup/index.html). \]\]>https://observablehq.com/blog/data-educators-froehlich https://observablehq.com/blog/data-educators-froehlichWed, 12 Jul 2023 00:00:00 GMT_Data Educator Stories explores the practices and projects of educators who use Observable in the classroom. The goal of this series is to grow our collective knowledge and build connections between the creative work of individuals and the broader education community._ [Jon Froehlich](https://www.cs.washington.edu/people/faculty/jonf) is a computer science professor using Observable to teach Intro to Data Visualization for non-majors at the University of Washington. Froehlich leads by example; sharing his own curriculum and personal projects while encouraging students to create and share socially-minded interactive visualizations. * * * **How do you use Observable in the classroom? Is there a specific course where it adds value?** I run [Intro to Data Visualization](/content/collection/@jonfroehlich/cse412-spr23/index.html) on Observable and I have over a hundred students each semester. I'm a computer science professor, but this is a non-majors course for fourth-year students interested in data science and [data visualization](/content/learn/what-is-data-visualization/index.html). Some come from statistics or math, but many others come from humanities, engineering, science, or other areas. We have a 100-level programming prerequisite, so we're starting with students who have a variety of technical skills and we support them in learning programming and building interactive data visualizations. Students start by learning how to sketch visualizations on paper before we graduate to programming; first in p5.js and then we [transition to Observable](/content/collection/@jonfroehlich/intro-to-p5-js-in-observable/index.html). [iframe](/content/embed/@jonfroehlich/using-p5js-in-observable?cells=doodleExample/index.html) _This cell is part of the transition from p5.js to Observable. Click on the canvas and move your mouse to change the brush attributes._ Students can fork my notebooks and follow along live as I'm coding in class. If they get lost, they can go back to my original notebook rather than trying to follow me on a shared screen, which is easier and more accessible. **What is something that is only possible in Observable that you find critical to your teaching?** Having a living environment on Observable that lets you live code with students is really powerful. You get to see mistakes happen. You get to see debugging happen. Class time is a limited learning opportunity, and since students have their own forked notebooks, they can track back to my original so the content lives on. For their course projects, students can also work collaboratively on code — just like they do in Google Docs. Collaborating like that is super powerful. Forking, which allows me to quickly create new materials, is the basis of a lot of my lectures. Beforehand, I’ll create two notebooks: [a skeleton version](/content/collection/@jonfroehlich/cse412-spr23/index.html) and a fully complete version with the answers. During class we work through the skeleton version together. After the lectures, I will publish the full version. **You have taught data visualization using both private and public notebooks. Can you explain your philosophy about sharing your own work?** I'm putting the time and investment into making good teaching materials, I don’t have any problem sharing them more broadly. That mentality was reinforced when we shifted our learning to online environments more generally in the pandemic. But my personal pedagogical take is that I want to make as much of an impact as I can. [Intro to Vega-Lite curriculum](/content/collection/@jonfroehlich/intro-to-vega-lite/index.html) and [Cartographic Visualization in Vega-Lite](/content/collection/@jonfroehlich/cartographic-visualization-in-vega-lite/index.html) are resources I’ve worked on recently. [iframe](/content/embed/@jonfroehlich/intro-to-interaction-in-vega-lite?cells=filterTempsExample/index.html) _Part of a sequence that allows students to explore and build on an example, adding filters and interactivity._ I also make mistakes, of course. That's definitely a risk to working in public but it’s also a benefit. Recently, a random stranger commented on one of my notebooks and it’s great that I get prompted to check in on my materials and to learn as well. **How do students submit their assignments with Observable?** Our assignments are notebooks. We don’t need to make the leap to embedding — students are creating a webpage in Observable already. There's no friction. Early in the quarter everything is unpinned and open while we’re teaching. For the final, we teach them to unpin cells for particular types of content so the focus is on the interactivity. **What kind of data do your students work with in Observable? Where does it come from and how does it help you teach?** I believe deeply in personalized learning, where students are compelled on their own learning journey. The concept for their course project is explanatory interactive data viz. Students write a story about a concept with societal impact or scientific importance, like global food security or climate change; trees and social wellbeing as an example. They find data that corresponds to that pitch and explain why it's a data set with integrity, then submit an interactive notebook in teams of three or four. This student's work explores the potential for solar power. [iframe](/content/embed/@jonfroehlich/sunshine-hours-in-the-united-states?cells=avgDailySunlightMap/index.html) _Click and drag over the graph on the left to select the range of average daily sunlight and the corresponding area on the map._ Observable is perfectly aligned to allow them to interact with the datasets as they are creating, to more easily create visuals, and then to create the interactives. **What are you working on now - academic or otherwise?** A big research project that my lab is working on is called [Project Sidewalk](https://sidewalk-sea.cs.washington.edu/). Consumers have access to road networks; governments have access to road networks. Mapping companies know where the roads are but they don't know where the sidewalks are, and none of these systems know the condition of sidewalks or if they're accessible to people with disabilities. Project Sidewalk allows us to find and analyze all the sidewalks in the world using computer vision and crowdsourcing. So it's a big data problem. [iframe](/content/embed/@jonfroehlich/live-project-sidewalk-stats?cells=psOverallStatsMap/index.html) _An Observable dashboard provides data about Project Sidewalk, including the areas explored, kilometers covered, and labels from contributors. This viz may take time to load._ \]\]>https://observablehq.com/blog/june-release-recap https://observablehq.com/blog/june-release-recapThu, 29 Jun 2023 00:00:00 GMTThis month’s new platform updates improve Enterprise workspaces and make [data wrangling](/content/learn/what-is-data-wrangling/index.html) easier. Read about these releases below and keep tabs on new features as they happen with our [release notes](/content/@observablehq/release-notes/index.html). ### Store files in your own S3 bucket Using private cloud storage for file attachments allows you to comply with internal security protocol and best practices when creating data visualizations with company or customer data. Now, if you have an Enterprise tier workspace, you can use a private S3 bucket to store file attachment data in Observable. Head to our [docs](/content/@observablehq/file-attachments#cell-1069/index.html) to learn more about private file storage. ### Convert Chart cell to Observable Plot code One of the benefits of Observable is that you can choose between no- and low-code charts, depending on your data work at hand. Now, in the [Chart cell](/content/@observablehq/chart-cell/index.html), you can convert your chart into [Observable Plot code](/content/plot/index.html). This means that you can quickly build a baseline chart in Chart cell and then convert it to code for unlimited customization. If you’re looking to up-level your Observable Plot skills, this is also a great way to interactively learn the Observable Plot API. You can [learn more about Observable Plot in the docs](/content/plot/index.html). ### Unlock insights in Chart cell Gain new insights in a fraction of the time with our recent Chart cell updates. Previously, non-linear positional and color scales were only available in Observable Plot, but they are now available in [Chart cell](/content/@observablehq/chart-cell/index.html). The Chart cell user interface has also been refreshed to emphasize the most commonly used fields. Reducers can be helpful for building certain charts, but since they are infrequently used we moved them into the 𝚺 menus that are available above each field. We also made it easier to identify the fields you've overridden. Pro tip: you can also quickly navigate the fields using the Tab key and cycle through values using the arrow keys. ### Conclusion We want to hear what you think about these new releases. Share your feedback or any additional feature ideas on [the feedback repo](https://github.com/observablehq/feedback). \]\]>https://observablehq.com/blog/student-data-jam-sp23-unca-data-dynamically https://observablehq.com/blog/student-data-jam-sp23-unca-data-dynamicallyFri, 16 Jun 2023 00:00:00 GMTObservable hosted our second [Student Data Jam](/content/@educators-network/data-jam-sp23?collection=@educators-network/data-jam-spring-23/index.html) this spring and the competing teams did not disappoint. The winner of this competition is Data Dynamically, from the University of North Carolina, Asheville. Eric Dunbar, Jacob Mello, and Asher Nejezchleb explored national college graduation rates and those of individual states by combining the contest datasets with additional data from the Integrated Postsecondary Education Data System (IPEDS). They used Bayes' Theorem to explore this data more deeply according to demographics like sex and race. Observable judges, [Robert Harris](/content/@trebor/index.html) and [Aaron Dennis](/content/@aaronkyle/index.html), selected Data Dynamically as the winning team because of their thorough data analysis and striking hero visualization. They awarded the team extra points for storytelling as the team provide context for their conclusions and detailed future questions. The team also submitted early for bonus points, which is no easy task during a busy semester. [iframe](/content/embed/5a5a00449cc5ffc4?cells=viewof+sex%2Cviewof+race%2Cviewof+mapView%2Cviewof+rate%2Ccalculated_rate_cell%2Chero_viz_cell/index.html) _Data Dynamically’s winning hero visualization is an interactive choropleth map that displays enrollment and graduation data broken down by sex and race._ * * * **What skills did your team bring to the project and how did they come together when creating your notebook?** **Eric Dunbar (ED):** Everyone on our team had started learning Observable in our _Dynamic Data Visualization_ class. The timing for the competition lined up with the final project, so our professor gave us the option to build a team and compete in the Data Jam. We decided that it would be a great opportunity to show off what we've learned in class and also get a good grade on the final. Jacob spent the most time on the math part, while Asher and I bounced between the different functions and data processing that we needed to do in JavaScript. And then, of course, we were all learning D3 and Observable in class. * * * **How was building with Observable helpful to your team?** **Jacob Mello (JM):** I have a lot of experience in R and a little bit in Python when it comes to data science. I thought maybe I would have to do all the data processing and then throw my results into a notebook, but it turns out we were able to do it all in Observable. Even without a programming background, I could see how I could have done this in Observable - it just would have taken much longer. **ED:** We made our hero visualization using the D3 library because we had already learned D3’s map capabilities in class. The approach we learned enabled us to make it very customizable; we could do a bunch of different things at once and we didn’t have to do it completely from scratch because we had this fantastic library. > I thought maybe I would have to do all the data processing and then throw my results into a notebook, but it turns out we were able to do it all in Observable. Jacob Mello, Student Data Jam winner **Asher Nejezchleb (AN):** My philosophy for a project like this is to only go as deep as you need to. We needed the visualization to do what we needed it to do; and if there's already a tool that does what you need it to do, don't look further than necessary. * * * **What was the most surprising part of your experience? Did you have any “a-ha” moments with the data?** **AN:** One of the a-ha moments was definitely, “Oh wait, I can put functions as plot parameters instead of just constants?” We had been taught how to use functions in our database class and that's where I got the idea that maybe I can just put a function in here - and it worked. **JM:** Two big moments leap out to me. The first involved our IPEDS data. We had imported it separately and split it into different tables; thankfully Asher and Eric handled combining and presenting that in Observable in an awesome way. [iframe](/content/embed/5a5a00449cc5ffc4?cells=viewof+state%2CTtest_for_state%2CTtest_data/index.html) _Select a state to highlight its T-score for graduation rate - a measure of how far each state is from the national average._ **ED:** I had some late-night realizations that something in the data was not what we thought it was, and I had to come up with new calculations to explore that new idea. I think the platform worked well in that way. Jacob and I had some long conversations; we spent a good couple hours talking about just different ways we could approach the data. **JM:** That led us to the second realization; our findings and conclusion. We only had statistics from the national level, then we had the idea to look at the state level to see if there were any significant differences. We considered listing the ten statistically best states and the ten statistically worst states and thought from there maybe we can infer what's causing the difference. But our conclusion was that they're all from different populations. There is no unified United States; no average that is meaningful. And that was definitely a surprise. * * * **How did working in Observable compare to other tools you’ve used?** **ED:** I definitely enjoyed Observable a lot more than some of the other notebook-based platforms. In the middle of this competition, I had another project that required me to do some data analysis without using Observable. It was definitely interesting having to do things manually, knowing that Observable wasn't gonna be there for that. **JM:** Observable is amazing, especially for data calculations and visualization. There are a lot of quality-of-life tools that are really nice. Many of them are small things that you don't even think about until you don't have them. For example, I can just attach a .csv in Observable, but it gets tricky in R because if I move the data or alter it in any way, then everything breaks. It doesn’t break with Observable, and not only that but you can easily copy usage code to your clipboard. You can just control-v and paste it and you don’t have to think about it, which is awesome. * * * ## **Building choropleth maps with Observable Plot** Want to build a choropleth map with Observable Plot? With the [geo mark](/content/plot/marks/geo/index.html), that’s possible in just a few lines! Copy the snippet below into a JavaScript cell in an Observable notebook, or fork our example from the [Plot gallery](/content/@observablehq/plot-choropleth/index.html). Replace **spatial\_data** and the values accessed for the fill color with your own to get started. ``` Plot.plot({ width: 975, height: 610, projection: "identity", marks: [Plot.geo(spatial_data, { fill: (d) => attribute_data.get(d.variable) })] }) ``` Check out Data Dynamically’s winning notebook, [The Relationship Between National Level Graduation Rates and State Level Graduation Rates by Race and Sex](/content/d/5a5a00449cc5ffc4/index.html) and learn about the benefits of using [Observable for education](/content/education/index.html). \]\]>https://observablehq.com/blog/data-educators-reagan-ahmad https://observablehq.com/blog/data-educators-reagan-ahmadWed, 07 Jun 2023 00:00:00 GMT_Data Educator Stories explores the practices and projects of educators who use Observable in the classroom. The goal of this series is to grow our collective knowledge and build connections between the creative work of individuals and the broader education community._ We caught up with two of the five instructors teaching sections of data visualization as part of the School of Information's Masters in Information and Data Science. They've used an Observable workspace to manage multiple sections of _Data Science 209_ for over 600 students since 2020. Course designer and lead teacher, [Andy Reagan](/content/@andyreagan/index.html), and lecturer [Mak Ahmad](/content/@makahmad/index.html) explained how they use Observable to teach [data visualization](/content/learn/what-is-data-visualization/index.html) to classes across semesters. * * * **What is something only Observable does that you find critical to your teaching?** **Andy Reagan (AR):** Instructors work from shared course content on Observable, allowing us to collaborate and make universal updates more quickly than through our teaching platform. A single version of our core teaching materials keeps our projects, assignments, and submission process consistent across classrooms. I helped former instructor and course designer [John Guerra-Gomez](/content/@john-guerra/index.html) develop the core framework for the course that all our teachers use, and we chose to use Observable because it's a really easy way to share content - for students and teachers. **Mak Ahmad (MA):** Observable is also really easy to demo in class. I can instantly give students a hands-on opportunity to learn a concept by forking a notebook, having them edit it, and presenting their findings. Students using Observable can easily share their assignments with me for grading, and other students for peer review. [iframe](/content/embed/@berkeleyvis/separate-order-and-align?cells=viewof+options,chart/index.html) _An example of an explainable demonstrating the concepts of separate, order, and alignment in JavaScript. See the code in_ [_this notebook_](/content/@berkeleyvis/separate-order-and-align/index.html) _and learn more in the course’s_ [_Explainables collection_](/content/collection/@berkeleyvis/explainables/index.html) _._ **AR:** We have five or six private orientation notebooks right in Observable that walk students through the course. We have recorded videos of some of the instructional content that include JavaScript demos in Observable, and we link to the demo notebooks for students to fork and follow along. I think one of the best uses for the platform is that it enables students to work with the examples we provide immediately, right in a live document. We provide the instructional content, assignment descriptions, and assignment data right in Observable, and then students submit their notebooks as assignments. **How do your students work with data in Observable? Where does it come?** **MA:** We provide the students with data, like [this sentiment analysis dataset](https://www.kaggle.com/datasets/kazanova/sentiment140), for assignments during the semester, so they can focus on building their visualization skills. The final project for Data Science 209 requires them to build a data-based narrative to tell a story with data visualization. By that time, they are welcome to use any data that they’d like, because they’ve already learned how to work with it. **What have you learned from students on their data journey?** **MA:** Regardless of the tool they use, data visualization empowers and enables students to tell amazing stories. I have learned a lot from a diverse set of projects based around mental health in tech; environmental issues like wildfires and disasters; sports, including Formula 1 and UFC; and the intersection of economy and nutrition - foodflation. [iframe](/content/embed/9ef3034e9ee8b8c1?cells=chart/index.html) _This student-created visualization animates the percentage of female executives at companies over time._ **What are you working on now - academic or otherwise? What’s next?** **MA:** I am currently very happy teaching data visualization at UC Berkeley and UC Davis while working on challenging technical industry problems, such as semantic extraction from video. Post- COVID, it’s more important than ever that university faculty show and discuss the applications of classroom exercises in real-world industry scenarios. Bridging the gap between academia and industry helps make the purpose of making visualizations more real to our students. **AR:** I recently gave a talk on Observable at the Five College Datafest. I talked to the participants and faculty engaged in the competition about what features are available, what they do, and I built a demo in real time. In twenty minutes, we had a dashboard talking to an API with all the JavaScript pieces in Observable; including a slider, radio buttons, and interactivity. You could build a dashboard like this with another tool, but it would have been much harder. Now that you have the JavaScript runtime setup in Observable, you're writing JavaScript and talking directly to a Python API, so you can really extend it and you won't really run into any roadblocks. You have a full feature set with all of these powerful inputs that is reasonably easy to build. I think there's something to be said for that. It's easy to build a “Hello, World!” example that looks nice, and a lot of tools have that, but Observable is really extensible and scales beyond that. \]\]>https://observablehq.com/blog/may-release-recap https://observablehq.com/blog/may-release-recapMon, 05 Jun 2023 00:00:00 GMTWe recently launched platform updates that improve Enterprise workspaces and make [data wrangling](/content/learn/what-is-data-wrangling/index.html) in the Data Table cell easier. Now, Enterprise accounts can better manage their workspaces with audit logs. With Derived Columns, we’ve reduced notebook clutter and hassle so you can simplify your workflow. Let’s explore these updates in more detail. ## Gain visibility into your Enterprise workspace with audit logs Managing your Enterprise workspace in Observable is now simpler with [audit logs](/content/@observablehq/audit-logs?utm_medium=blog&utm_campaign=icymi_052023/index.html) that capture notebook activity across your workspace by tracking key events, like notebook creation, editing, and sharing. Workspace owners can download 180 days of workspace activity in either CSV or JSON formats. Our [documentation](/content/@observablehq/audit-logs?utm_medium=blog&utm_campaign=icymi_052023/index.html) covers the full list of events captured in audit logs. Enable audit logs today or [learn more about Enterprise](/content/pricing?utm_medium=blog&utm_campaign=icymi_052023/index.html) workspace features. ## Wrangle data with Derived Columns It’s now easier to wrangle data in the Data Table cell with the launch of Derived Columns. Previously, if you wanted to modify or generate data in the Data Table cell, you’d have to create JavaScript cells before or after the Data Table cell. This created an extra step in the workflow and an extra hassle, especially when working with messy CSV data. You can now get rid of the additional clutter in the notebook with Derived Columns. If you create a cell from a file attachment or JavaScript array, this feature will provide a code editor directly in the Data Table cell so you can write JavaScript expressions to structure and clean your data. Learn more about [Derived Columns](/content/@observablehq/data-table-cell#columnDerivation?utm_medium=blog&utm_campaign=icymi_052023/index.html) or [head to a notebook](/content/team/new?utm_medium=blog&utm_campaign=icymi_052023&utm_source=feature_announcement/index.html) to give it a try. ## Conclusion That’s it for this month’s platform updates. Stay tuned to [our release notes](/content/@observablehq/release-notes?utm_medium=blog&utm_campaign=icymi_052023/index.html), because we have some big updates on deck for next month. In the meantime, if you have feedback on these features, or any additional feature ideas, we want to hear from you on [the feedback repo](https://github.com/observablehq/feedback). \]\]>https://observablehq.com/blog/linsley-observable-plot https://observablehq.com/blog/linsley-observable-plotThu, 18 May 2023 00:00:00 GMTOpen source is at the core of Observable. Tanner Linsley, creator of the TanStack (over 20M monthly downloads between all libraries), is no stranger to open source. In addition to being an experienced open source contributor and Observable community member, Linsley is co-founder and VP of UI / UX at [Nozzle](https://nozzle.io/),  an enterprise SEO monitoring tool. When Linsley needed a JavaScript library for building visualizations of various data models in Nozzle, from keyword management to analytical dashboard design, his tool of choice was [Observable Plot](/content/plot/index.html). ## One of many examples: giving users a simple way to visualize their keyword configurations For this particular example, Linsley’s question was a simple one, but not so simple to solve: How can he give Nozzle users a powerful way to visualize their unique keyword phrase permutations in Nozzle’s keyword manager? Before using Plot, Linsley was limited to imperative visualizations libraries that he said, "offered pretty, but ultimately unwieldy visualizations," that required manual intervention or simply resulted in bad data density and user experience. Not only did these libraries incur a less-than-optimal developer experience, but the resulting visualizations were confusing or overwhelming to Nozzle users. ## Using Plot to explore complex, real-time data Built on top of [D3](https://d3js.org/), Observable Plot is a best-in-class JavaScript library for exploratory [data visualization](/content/learn/what-is-data-visualization/index.html). According to Linsley, “Plot allowed us to take data of any granularity and quickly create meaningful visualizations that give the user the information they need to understand their keyword configurations.” At the enterprise level, these keyword configurations can get quite complex. They’re driven by multiple sources like phrases, locations, languages, templates, devices, and location. When thousands of these keywords are generated, it can be difficult to discern how each configuration is contributing to the overall generation. Linsley explains, “With Plot’s ability to drastically change visual output from surprisingly minimal changes to code, we were able to explore the data efficiently and arrive at the right final visualization for each of the configurations we needed. The example below, while still simple, breaks down keyword sources by various facets so our clients can more easily manage which keyword configurations are contributing the most to their usage.” Observable Plot allowed Linsley to create a dash that allowed clients to more easily manage which keyword configurations contribute the most to their usage. ## The outcome: simple, expressive data viz that’s easy to tweak Since picking up D3 almost a decade ago, Linsley said,  “Plot is easily the simplest, most powerful data visualization tool I’ve used since I started using D3.” He emphasized both ease of implementation, as well as an ability to build a better product for Nozzle users, saying, “The entire library frees us of implementation details while still affording extreme granularity when required. At the bottom line, it has directly increased our speed and ability to provide our users with better information concerning every aspect of our product.” > With Plot’s ability to drastically change visual output from surprisingly minimal changes to code, we were able to explore the data efficiently and arrive at the right final visualization for each of the configurations we needed. > > \- Tanner Linsley, Co-founder and VP of UI / UX at Nozzle ## An intuitive grammar of graphics Linsley had tried using different visualization libraries but found that they slowed down his analysis and were needlessly complex. “Many higher-level visualization libraries claim ‘declarative’ configuration, but fail to implement grammar of graphics properly. This means that you eventually end up writing a lot of declarative, but unnecessary, implementation details to achieve the visual output you want,” explained Linsley. He continued, "It slows you down and makes it difficult to build solutions that can be as dynamic as the data you want to explore. In the end, you may not arrive at the right solution because of a time constraint or simply due to the pain of implementation details.” Plot is purpose-built to be an intuitive tool for beginners and data viz experts alike. “Plot is the first tool I've used that implements grammar of graphics in a way that allows me to explore data efficiently with the right level of abstraction and customization,” said Linsley. He continued, "TypeScript support makes it so much more enjoyable and things are apparently only going to get better!” ## Interested in learning more about Observable Plot? There are several free resources to jumpstart your learning of [Observable Plot](/content/plot/index.html). Check out [Learning Observable](/content/@observablehq/learning-observable-observable-plot?collection=@observablehq/intro-to-observable/index.html), or watch a [video](https://www.youtube.com/watch?v=pgga4jWda4c). We also offer full courses - learn more by connecting with other data viz creators on [Slack](https://join.slack.com/t/observable-community/shared_invite/zt-1icwex7uf-ovIAS0PcuJ~ZgBToSyUs8w?) and [Mastadon](https://vis.social/@observablehq). Or, ask your questions on the Observable [forum](https://talk.observablehq.com/). \]\]>https://observablehq.com/blog/april-release-recap https://observablehq.com/blog/april-release-recapWed, 03 May 2023 00:00:00 GMTLast month we launched private embeds, made search _a lot_ faster, and improved the experience of using Observable. Now, you can securely share private embeds, and search for content within your team or the Community to get more accurate results — 10x faster. Plus, new Enterprise features help larger teams better organize their work and the Data Table cell is easier to use. Let’s explore what these new features mean for your workflow: ## Share private notebooks with secure, private embeds [Data visualization](/content/learn/what-is-data-visualization/index.html) is most impactful when it goes from your browser to teammates in a way that can foster discussions and create shared understanding. Observable makes it easy for you to embed notebooks and supports a variety of embed types, including iFrame, JavaScript, and React. We’re continuing to improve this feature by making it simpler and more secure. First, to increase the security of private notebook embeds, we’ve created notebook keys. A [Notebook Key](/content/@observablehq/api-keys?utm_medium=blog&utm_campaign=icymi_042023&utm_source=feature_announcement/index.html) only has access to a specific version of a single notebook and the resources, such as database connections and secrets, used by the notebook. This key type is more granular than our existing API keys, which are scoped to the workspace rather than a specific notebook. To create a notebook key, you will use our new one-click interface, which simplifies and accelerates the embed creation process. In this interface you’ll also notice that you can set a key expiration date, allowing you to create short-lived keys. Previously, iFrame embeds couldn’t access workspace secrets and databases, which prevented some teams from using embeds. Now, if you have a [Pro or Enterprise workspace](/content/pricing?utm_medium=blog&utm_campaign=icymi_042023&utm_source=feature_announcement/index.html) and use iFrame notebook embeds, those embeds can access the workspace secrets and databases that are referenced within the notebook. This makes notebook embeds accessible for more teams. Lastly, for Enterprise workspaces, we’ve developed the option to remove the _Powered by Observable_ attribution in iFrame embeds. This new functionality opens up an array of potential use cases. For instance, you can now add charts demonstrating KPIs, such as feature engagement or revenue from a specific product tier, to internal Notion pages. Embed visualizations in internal apps to keep your whole team on the same page. Or, use data visualizations to support visual storytelling in reports. Learn more about these updates, [how to embed Observable notebooks](/content/@observablehq/embeds?utm_medium=blog&utm_campaign=icymi_042023&utm_source=feature_announcement/index.html), and tips for [troubleshooting embeds](/content/@observablehq/troubleshooting-embedding?utm_medium=blog&utm_campaign=icymi_042023&utm_source=feature_announcement/index.html), in our docs. ## Find what you’re looking for 10x faster Discovering content on Observable, whether within your team workspace or within our Community, starts with the search bar. So, when we noticed search was slowing our users down, we knew we had to address it. Now, search returns more accurate results faster. Not just a little faster. 10x faster. Old search speed v. new search speed ## Manage team access and organize notebooks with Groups To successfully collaborate in Observable, team members need to have access to the correct notebooks. Sharing notebooks with teams in your organization is now easier with [Groups](/content/@observablehq/release-notes?utm_medium=blog&utm_campaign=icymi_042023&utm_source=feature_announcement#cell-2998/index.html). This feature, available for Enterprise workspaces, allows you to define a group in Observable, such as a project team or a department. Once added to a group, users can share notebooks with their group or other individuals in a workspace. As users are added to your Observable workspace, simply add them to the right groups and all the correct notebooks will automatically be shared with them. ## Select and filter Data Table cell values without writing conditions By using the Data Table cell, you can, as the name suggests, quickly take your data from a data source and put it into a table. If you are familiar with the Data Table cell, then you likely know that it allows you to filter your data. With this update, there’s a faster and easier way to select and filter values. Simply click on a cell in the table and you’ll receive a menu of filter options relative to the data type of the column. Using the cell value filter improves your workflow in two ways. First, it saves time that you’d otherwise spend writing the conditions manually. Second, it’s more precise than using the column header’s Summary Charts. Since we are talking about the Data Table cell, don’t forget that our recent [Chart cell](/content/blog/chart-cell?utm_medium=blog&utm_campaign=icymi_042023&utm_source=feature_announcement/index.html) feature makes it much, much easier to build data visualizations based on the data you’ve selected in the Data Table cell. If you haven’t yet tried Chart cell, what are you waiting for? [Log into your account](/content/team/new?utm_medium=blog&utm_campaign=icymi_042023&utm_source=feature_announcement/index.html) and see for yourself how fast you can go from data to visualization. Need a refresh of what you can do with the Data Table cell? The [docs are right this way](/content/@observablehq/data-table-cell?utm_medium=blog&utm_campaign=icymi_042023&utm_source=feature_announcement/index.html). ## Conclusion We have a lot of other exciting updates on deck for next month. To stay informed as features roll out, [visit our release notes](/content/@observablehq/release-notes?utm_medium=blog&utm_campaign=icymi_042023&utm_source=feature_announcement/index.html). As a reminder, we are always looking forward to hearing feedback from users, so please share your thoughts on the [feedback repo](https://github.com/observablehq/feedback). \]\]>https://observablehq.com/blog/r-to-javascript-shapiro https://observablehq.com/blog/r-to-javascript-shapiroTue, 25 Apr 2023 00:00:00 GMTTanya Shapiro is a self-described "part data geek, part business detective." As an independent data consultant, she helps businesses unlock the value of their data. Shapiro specializes in [data visualization](/content/learn/what-is-data-visualization/index.html) and dashboard design, streamlining reports, delivering [business intelligence](/content/learn/what-is-business-intelligence/index.html) solutions, and more.  She applies a data product management mindset to help clients tackle a variety of problems within their analytics pipeline. After attending last year’s Observable Insight conference, Shapiro’s interest in using Observable to work with data was piqued. Since then, she’s created some of the most unique and [popular notebooks on Observable](/content/@tashapiro/index.html). Shapiro’s notebooks on Observable are popular, in this author’s opinion, because they do a few things. First, Shapiro uses Observable to its full potential — leveraging the ability to pull in multiple data sources fast and get from [data exploration](/content/learn/what-is-data-exploration/index.html) to a [data visualization](/content/learn/what-is-data-visualization/index.html) that tells a compelling story in a few clicks. Second, her notebooks are… fun. Read on to learn more. [iframe](/content/embed/3ac7dfcbdcb1dd2a@248?cells=map,css/index.html) | | | --- | | For Tanya Shapiro’s map of **RuPaul's Drag Queens** around the world above, data was scraped from [Drag Race Wiki](https://rupaulsdragrace.fandom.com/wiki/RuPaul%27s_Drag_Race_Wiki). Shapiro says, “It's wild to see how RuPaul's footprint grows as the show adds different international franchises. There are over 350 queens on this map from more than 10 franchises!” | * * * **How did you get started with data visualization? How did you start using Observable?** I came into the data visualization world in large part thanks to the online R community. Over the past few years, I’ve met some exceptionally talented people — both online and in real life — including one of my data viz heroes, [Allison Horst](/content/@allisonhorst/index.html). My curiosity about Observable was already piqued after attending the Observable Insight conference. I distinctly remember having that “oh wow” moment when [Maya Gans](/content/@mayagans/index.html) started talking about polyglot workflows with R and Observable. I was inspired by many of the talks at the event, and motivated to find different ways to build my own interactive visuals. Then there was this fortuitous overlap of events. Allison and I connected the following week to hang out and geek out about data viz. We talked about our respective journeys, the power of learning through community, and the different tools we work with. With her encouragement, I started experimenting with Observable. Since we both share an R background, [her ability to translate JavaScript and Observable concepts](https://viewablehq.com/cms-preview/resources/cheatsheets/data-wrangling-comparisons) has been very helpful. She has a gift for making complex data topics accessible to beginners. I still bug her with my Observable questions today! > _When I found out about Observable notebooks, I was ecstatic — I didn’t have to worry about the setup — I could start experimenting directly in the notebook and see the outputs instantly._ > > \- Tanya Shapiro, Independent Data Consultant **I see you have a Drag Queens Netwerk Diagram on Observable,** [**as well as on your site, built using other tools**](https://www.tanyashapiro.com/rupauls-drag-queens) **. Can you talk a bit about the process of working on Observable vs. other tools?** Yes, I’m a big fan of RuPaul’s Drag Race! I think it’s always easier to learn new things when you’re working with a topic you’re interested in. [iframe](/content/embed/f3941ff4743f26e3?cells=chart,css/index.html) | | | --- | | For Tanya Shapiro’s Drag Queens: Netwerk Diagram above, she modified code from [Guillermo Garcia's notebook](/content/@garciaguillermoa/force-directed-graph/index.html) and [**Raven Gao's notebook**](/content/@ravengao/force-directed-graph-with-cola-grouping/index.html) to experiment with network diagrams. Netwerk Diagram shows drag queens who were featured on different franchises and seasons of the TV reality competition series, **RuPaul's Drag Race**. | Last year, I used it as an opportunity to learn more about web scraping and [data wrangling](/content/learn/what-is-data-wrangling/index.html) with R (rvest). I might be biased because I’ve been working with R for a few years, but I think data wrangling and pre-processing in R is easy thanks to libraries like dplyr. I also created a couple of visuals in R to explore the data, including a static network diagram with ggplot2 to show how all the contestants were connected throughout the different seasons. This year I’m pushing myself to create more interactive visual content. I’ve tried a few different libraries in R to bring some of my visuals to life, but I started noticing a theme: everything comes back to JavaScript. A lot of the interactive visual libraries in R are actually JavaScript wrappers. If I wanted to customize something beyond the wrapper’s ability, I’d end up having to hack a solution by injecting more JavaScript in R. Which led me to the conclusion - I should probably try these ideas out with ACTUAL JavaScript! I think one of the biggest hurdles for a JavaScript beginner to get started is the setup process (eg. download an IDE, install node.js, create html and css files, etc). When I found out about Observable notebooks, I was ecstatic — I didn’t have to worry about the setup — I could start experimenting directly in the notebook and see the outputs instantly. One of the first ideas I wanted to try out was a revamped version of my Drag Race network diagram. I recycled the data I wrangled from my R project — and was immediately impressed by how easy it was to upload csv files in Observable. As a JavaScript padawan, I found the D3 coding part behind the graphic a little bit trickier. But luckily I had a lot of help via the Observable community. I’ve learned a lot from exploring other people’s notebooks, and when I’m stuck, there’s always someone on the  Observable Slack channel willing to help me troubleshoot. > _I think it’s always easier to learn new things when you’re working with a topic you’re interested in._ > > \- Tanya Shapiro, Independent Data Consultant For the grand finale, I tested my graphic in R Quarto. I often use Observable notebooks as my data viz playground. When I’m satisfied with the end product, I like translating it to Quarto, which supports Observable and R code. I can do the data wrangling in R, pass the data into an Observable code chunk to create beautiful interactive graphics, and then knit it all together to produce the html file. I’m excited to play around with these workflows more. R and Observable is easily my favorite dynamic duo. **You've partnered with groups like** [**R Ladies**](https://www.rladies.org/) **and** [**Women Who Code**](https://www.womenwhocode.com/) **to share your knowledge of data viz design and to help others develop their technical skills. What advice do you have for people interested in getting started with data viz?** We all start somewhere, the trick is to just start… and then keep going! I’d also add don’t be afraid to experiment and explore new ideas even if you think they’re awful, you’ll learn something new in the process, I promise. The advice I give myself a lot these days now that I’m picking up a new coding language: be patient with yourself. Progress isn’t always linear, I’m far off from where I want to be, but I’m falling in love with the journey. **How can people learn more about your work and ask questions about working with you?** Learn more about my work and [reach out about working with me here.](https://tanyaviz.com/) Thank you! \]\]>