Defining Discourse Data Types

Harwood Plains
Harwood Plains

After learning how to authenticate users on a Remix app with Discourse, I had to try displaying Discourse topics in the app.

Pulling in the data was easy:

  const headers = new Headers();
  if (apiKey) {
    headers.append("Api-Key", apiKey);
    headers.append("Api-Username", "system");
  }

  const response = await fetch(`${discourseBaseUrl}/latest.json`, {
    headers: headers,
  });
  const latestTopics= await response.json();

The tricky part was defining the data’s Typescript types.

I started by trying to define the types without looking closely at the structure of the JSON returned from Discourse. I was guessing about what might work, then fixing any issues that were reported by the Typescript compiler – more focused on getting rid of Typescript warnings than on creating useful data types.

Instead of trying to clean up that mess, I’ll start again with a new route:

$ touch app/routes/topics.tsx

Here’s a simple route that makes an API request to get the Discourse Latest Topics list. It then displays the usernames of users who have posted in the list’s topics:

// app/routes/topics.tsx

import { json, redirect } from "@remix-run/node";
import { useLoaderData } from "@remix-run/react";

export const loader = async () => {
  if (!process.env.DISCOURSE_BASE_URL || !process.env.DISCOURSE_API_KEY) {
    return redirect("/");
  }
  const discourseBaseUrl = process.env.DISCOURSE_BASE_URL;
  const apiKey = process.env.DISCOURSE_API_KEY;
  if (!discourseBaseUrl || !apiKey) {
    return redirect("/");
  }

  const headers = new Headers();
  headers.append("Api-Key", apiKey);
  headers.append("Api-Username", "system");

  const response = await fetch(`${discourseBaseUrl}/latest.json`, {
    headers: headers,
  });
  const latestTopics = await response.json();

  return json({ latestTopics });
};

export default function Topics() {
  const { latestTopics } = useLoaderData<typeof loader>();
  const users = latestTopics.users;
  console.log(`latestTopics: ${JSON.stringify(latestTopics, null, 2)}`);
  return (
    <div>
      <h1>Latest Topics</h1>
      <ul>
        {users.map((user) => (
          <li key={user.id}>{user.username}</li>
        ))}
      </ul>
    </div>
  );
}

This is where the fun starts 🙂

Typescript is warning: “Parameter ‘user’ implicitly has an ‘any’ type.” The warning can be removed with:

{users.map((user: any) => (
  <li key={user.id}>{user.username}</li>
))}

but giving the data the any type defeats the purpose of using Typescript.

For my own reference, here’s the JSON that’s being returned from the API request:

latest.json 
// http://localhost:4200/latest.json

{
  "users": [
    {
      "id": -1,
      "username": "system",
      "avatar_template": "/images/discourse-logo-sketch-small.png",
      "admin": true,
      "moderator": true,
      "trust_level": 4,
      "assign_icon": "user-plus",
      "assign_path": "/u/system/activity/assigned"
    },
    {
      "id": 1,
      "username": "simon",
      "avatar_template": "/user_avatar/127.0.0.1/simon/{size}/33_2.png",
      "primary_group_name": "customer_support",
      "flair_name": "customer_support",
      "flair_group_id": 41,
      "admin": true,
      "moderator": true,
      "trust_level": 4,
      "assign_icon": "user-plus",
      "assign_path": "/u/simon/activity/assigned"
    },
    {
      "id": 7,
      "username": "scossar",
      "avatar_template": "/user_avatar/127.0.0.1/scossar/{size}/57_2.png",
      "admin": true,
      "trust_level": 1,
      "assign_icon": "user-plus",
      "assign_path": "/u/scossar/activity/assigned"
    }
  ],
  "primary_groups": [
    {
      "id": 41,
      "name": "customer_support"
    }
  ],
  "flair_groups": [
    {
      "id": 41,
      "name": "customer_support",
      "flair_url": null,
      "flair_bg_color": "",
      "flair_color": ""
    }
  ],
  "topic_list": {
    "can_create_topic": true,
    "more_topics_url": "/latest?no_definitions=true&page=1",
    "per_page": 30,
    "top_tags": ["foo", "bar", "sci-fi", "aldis", "lessing"],
    "topics": [
      {
        "id": 5,
        "title": "Welcome to WP Discourse Development! :wave:",
        "fancy_title": "Welcome to WP Discourse Development! :wave:",
        "slug": "welcome-to-wp-discourse-development",
        "posts_count": 3,
        "reply_count": 0,
        "highest_post_number": 3,
        "image_url": null,
        "created_at": "2023-08-16T21:24:31.399Z",
        "last_posted_at": "2024-03-28T06:16:58.960Z",
        "bumped": true,
        "bumped_at": "2024-03-28T06:16:58.960Z",
        "archetype": "regular",
        "unseen": false,
        "last_read_post_number": 2,
        "unread": 0,
        "new_posts": 0,
        "unread_posts": 0,
        "pinned": false,
        "unpinned": true,
        "visible": true,
        "closed": false,
        "archived": false,
        "notification_level": 1,
        "bookmarked": false,
        "liked": false,
        "unicode_title": "Welcome to WP Discourse Development! 👋",
        "tags": [],
        "tags_descriptions": {},
        "views": 4,
        "like_count": 0,
        "has_summary": false,
        "last_poster_username": "system",
        "category_id": 4,
        "pinned_globally": true,
        "featured_link": null,
        "has_accepted_answer": false,
        "can_vote": false,
        "posters": [
          {
            "extras": "latest single",
            "description": "Original Poster, Most Recent Poster",
            "user_id": -1,
            "primary_group_id": null,
            "flair_group_id": null
          }
        ]
      },
      {
        "id": 462,
        "title": "Testing to see what happens",
        "fancy_title": "Testing to see what happens",
        "slug": "testing-to-see-what-happens",
        "posts_count": 31,
        "reply_count": 15,
        "highest_post_number": 31,
        "image_url": null,
        "created_at": "2023-11-04T09:31:36.753Z",
        "last_posted_at": "2024-03-28T05:28:25.844Z",
        "bumped": true,
        "bumped_at": "2024-03-28T05:28:25.844Z",
        "archetype": "regular",
        "unseen": false,
        "pinned": false,
        "unpinned": null,
        "visible": true,
        "closed": false,
        "archived": false,
        "bookmarked": null,
        "liked": null,
        "tags": [],
        "tags_descriptions": {},
        "views": 4,
        "like_count": 0,
        "has_summary": false,
        "last_poster_username": "system",
        "category_id": 4,
        "pinned_globally": false,
        "featured_link": null,
        "has_accepted_answer": false,
        "can_vote": false,
        "posters": [
          {
            "extras": null,
            "description": "Original Poster",
            "user_id": 1,
            "primary_group_id": 41,
            "flair_group_id": 41
          },
          {
            "extras": "latest",
            "description": "Most Recent Poster",
            "user_id": -1,
            "primary_group_id": null,
            "flair_group_id": null
          }
        ]
      }
    ]
  }
}

I hid that for a reason. It’s a lot of fields!

Discourse is a Ruby on Rails/Ember application. Its Ruby on Rails API returns JSON data that’s needed by the Ember front end. My app doesn’t need all that data.

These are the fields I’m interested in:

{
  "users": [
    {
      "id": -1,
      "username": "system",
      "avatar_template": "/images/discourse-logo-sketch-small.png"
    }
  ],
  "topic_list": {
    "more_topics_url": "/latest?no_definitions=true&page=1",
    "topics": [
      {
        "id": 462,
        "title": "Testing to see what happens",
        "unicode_title": "Testing to see what happens 👋",
        "slug": "testing-to-see-what-happens",
        "posts_count": 31,
        "created_at": "2023-11-04T09:31:36.753Z",
        "last_posted_at": "2024-03-28T05:28:25.844Z",
        "archetype": "regular",
        "views": 4,
        "like_count": 0,
        "last_poster_username": "system",
        "category_id": 4,
        "posters": [
          {
            "description": "Original Poster",
            "user_id": 1
          },
          {
            "description": "Most Recent Poster",
            "user_id": -1
          }
        ]
      }
    ]
  }
}

That reduces things to:

  • a users array (all of the users who are in the topic_list.topics.posters arrays)
  • a topic_list object that has a more_topics_url field (used to query for the next page of topics) and a topics array
  • a topics_list.topics array. It has fields for id, title, slug, etc, and a posters array.
  • a topics_list.topics.posters array

I’ll start by defining types for each of the objects that are found in the users, topics, and posters arrays:

interface User {
  id: number;
  username: string;
  avatar_template: string;
}
interface Poster {
  description: string;
  user_id: number;
}
interface Topic {
  id: number;
  title: string;
  unicode_title?: string;// only set if the title contains an emoji
  slug: string;
  posts_count: number;
  created_at: string;
  last_posted_at: string;
  archetype: "regular" | "private_message"; // a union type (archetype can be either "regular" or "private_message")
  views: number;
  like_count: number;
  last_poster_username: string;
  category_id: number;
  posters: Poster[]; // an array of Poster types
}

The JSON that’s returned for the Latest Topics list has the same structure as the data that’s returned for other Discourse topic lists. I’ll define a TopicList type that can be used for any of the app’s Discourse topic lists. (Somewhere along the way I seem to have decided to make a Remix/Discourse app):

interface TopicList {
  users: User[]; // an array of User types
  topic_list: {
    more_topics_url?: string; // only set if there's another page of topics
    topics: Topic[]; // an array of Topic types
  };
}

The TopicList type can be used to indicate the type that’s returned from the loader function:

  const response = await fetch(`${discourseBaseUrl}/latest.json`, {
    headers: headers,
  });
  const latestTopics: TopicList = await response.json();

  return json({ latestTopics });
};

export default function Topics() {
  const { latestTopics } = useLoaderData<typeof loader>();
  const users = latestTopics.users;
  const topicList = latestTopics.topic_list;
  const topics = topicList.topics;

Typescript is now aware that in the route’s component:

  • latestTopics has the type JsonifyObject<TopicList>
  • users has the type JsonifyObject<User>[]
  • topicList has the type JsonifyObject<more_topics_url?: string | undefined; topics: Topic[]>
  • topics has the type JsonifyObject<Topic>[]

The JsonifyObject<T> type is interesting. T is the object’s type. JsonifyObject comes from here: https://github.com/remix-run/remix/blob/0dc1da28cd78404fc1d836296e514adeb7c78aca/packages/remix-server-runtime/jsonify.ts#L63-L79.

My initial assumption was that JsonifyObject<T> was the return type of Remix’s json helper function, but values returned from a loader have the JsonifyObject<T> type even if the the json function isn’t used in the return statement. JsonifyObject<T> is being set as the type somewhere in the Remix code base. It takes care of generating types when an object is serialized to JSON in a loader function, then deserialized back into a JavaScript object on the client. For my purposes, it’s enough to know that it works.

Error handling

The type declaration in the loader function is overly confident:

const latestTopics: TopicList = await response.json();

What if the response from Discourse is an error?

That’s not really a Typescript issue. It can be handled before dealing with the data’s types by throwing a Response:

const response = await fetch(`${discourseBaseUrl}/latst.json`, {
  headers: headers,
});
if (!response.ok) {
  throw new Response("Failed to fetch topics", {
    status: response.status,
    statusText: response.statusText,
  });
}

Filtering data in the loader

With errors out of the way, it’s (reasonably) safe to assume that data sent to the client will match the TopicList type. The problem now is that a lot of data that won’t get used is being sent to the client.

That can be dealt with by filtering the data in the loader function:

const data = await response.json();

const latestTopics: TopicList = {
  users: data.users.map(({ id, username, avatar_template }: User) => ({
    id,
    username,
    avatar_template,
  })),
  topic_list: {
    more_topics_url: data.topic_list.more_topics_url,
    topics: data.topic_list.topics.map((topic: Topic) => ({
      id: topic.id,
      title: topic.title,
      unicode_title: topic.unicode_title,
      slug: topic.slug,
      posts_count: topic.posts_count,
      created_at: topic.created_at,
      last_posted_at: topic.last_posted_at,
      archetype: topic.archetype,
      views: topic.views,
      like_count: topic.like_count,
      last_poster_username: topic.last_poster_username,
      category_id: topic.category_id,
      posters: topic.posters.map((poster: Poster) => ({
        description: poster.description,
        user_id: poster.user_id,
      })),
    })),
  },
};

How safe is it to assume the returned data will match the defined types?

The frivolous answer is that for my case it’s safe enough.

It’s good to know where things could go wrong though. For example, calling data.topics_list.map instead of data.topic_list.map will trigger an internal server error:

topic_list: {
  more_topics_url: data.topic_list.more_topics_url,
  topics: data.topics_list.topics.map((topic: Topic) => ({

Trying to access a property that doesn’t exist won’t trigger an error, as long a no methods are called on the property. For example:

topic_list: {
  more_topics_url: data.topic_list.more_topics_url,
  topics: data.topic_list.topics.map((topic: Topic) => ({
    id: topic.id,
    foo: topic.bar,

Although it could cause problems if the user interface was dependent on topic.foo.

For an application more serious than “Simon’s Blog,” there are a few ways the data could be validated before sending it to the client.

Here’s a function that does a minimal check to confirm that data returned from Discourse is a valid TopicList:

// confirm that the data returned has a users
// array, and a topic_list object that contains
// a topics array

function isValidTopicList(data: any): data is TopicList {
  return (
    "users" in data &&
    Array.isArray(data.users) &&
    "topic_list" in data &&
    data.topic_list?.topics &&
    Array.isArray(data.topic_list.topics)
  );
}

isValidTopicList can be called in the loader function to trigger a 400 response if data isn’t a valid TopicList:

if (!isValidTopicList(data)) {
  throw new Response("invalid data returned", { status: 400 });
}

Another approach is to filter out invalid data, but still send a response to the client:

// confirm that a topic has an id, title, and
// slug of the appropriate types

function isValidTopic(topic: any): topic is Topic {
  return (
    "id" in topic &&
    typeof topic.id === "number" &&
    "title" in topic &&
    typeof topic.title === "string" &&
    "slug" in topic &&
    typeof topic.slug === "string"
  );
}

Then call it in the loader:

// filter out topics that don't have the required
// properties

topic_list: {
  more_topics_url: data.topic_list.more_topics_url,
  topics: data.topic_list.topics
    .filter(isValidTopic)
    .map((topic: Topic) => ({

Going beyond error handling, the filter pattern could be used to remove otherwise valid topics that don’t meet a specific criteria. The example below removes topics that don’t meet a like_count threshold. The function reminds me of The Structure and Interpretation of Computer Programs. I find that book’s exercises to be difficult, but it changed the course of my life when I was in my early 40s.

// filter out topics that don't meet a given
// like_count threshold (defaults to 2 likes)

function meetsTopicLikeCountThreshold(minLikeCount = 2) {
  return (topic: Topic): boolean => {
    return topic.like_count >= minLikeCount;
  };
}

Use it in the loader like this:

// not a lot of likes on my dev forum
// so I've set minLikeCount to 1

topic_list: {
  more_topics_url: data.topic_list.more_topics_url,
  topics: data.topic_list.topics
    .filter(isValidTopic)
    .filter(meetsTopicLikeCountThreshold(1))

This is fun stuff! But it’s taking me away from the headless WordPress project.

But young or old, the main barrier to finishing is starting other new and more interesting stuff. You have to sacrifice the attraction of new things, better things. I have a file on my computer called BESTSELLING IDEAS. Whenever I am writing and I have an idea that is WAY BETTER than the book I am writing I put it in the bestsellers file. This sates the desire to dump the mediocre crap I am writing in favour of some new holy grail of infinite possibility (how a lot of ideas look before you’re mangled and pawed them to death).

On Finishing: Robert Twigger

I might be able to compromise here. The WordPress/Remix app needs a comment system. Discourse could be used for that 🙂