Dealing With Paginated Data

(I’m working on a Remix app that uses Discourse as the back end for its comment system. It was going great until I got to the question of how to handle large amounts of paginated data. This post documents the process of figuring that out. Scroll to the bottom to see the answer. The rest of what’s documented here is probably only of interest to me.)

Searching the web for “Remix infinite scrolling” returned this Hacker News post from 2022 by Ryan Florence:

Remix co-author here. Infinite scroll/pagination is a great way to test the limits of a web framework!

Remix doesn’t ship an infinite scroll/pagination set of components so it’s up to apps to make sure that state is still there when the user clicks back. If the state is still there, Remix’s scroll restoration will work.

You could either manage your own global state for this, but I like to use “location state” which is our API into the browser’s built in `history.state`.

There are various ways to use it (declaratively, imperatively, etc.), but simplest way to explain is to imagine a “Load more” button that’s really just a link like this:

`<Link to=”?page=2″ state={{ data }} />`

Your Remix loader would load page 2 from the url search param and your component would concat that onto `location.state.data` (the previous entries from the initial location). This renders the old data and the new data together.

Location state, unlike typical “global state”, automatically persists across both refreshes and back/forward button clicks, but dies with the session, so scroll restoration will work as expected even for a refresh! Built in browser APIs tend to be a bit more resilient than storing stuff in application memory, they also keep your bundle smaller.

I don’t know where the demo is, but I helped somebody at some point implement this without needing to ship a global state container for server-supplied data. Just made a note to make an example and put it our repo 🙂

If we’re talking “load more” a much simpler way to do it is to consider how you’d do it old school with no JS. Just return all pages according to the search param, so “?page=3” would return all three pages. More kB over the network, but far easier to implement. There’s even a product reason to do it this way: when you load more you automatically update the comment counts/points of each entry, so maybe it’s worth it.

Great question!
https://news.ycombinator.com/item?id=30376689

That got me searching for the example code that he mentions in the post. I suspect it’s here: https://github.com/remix-run/examples/tree/main/infinite-scrolling.

Before going to far with the example code, I’ll copy the backend.server.ts file from https://github.com/remix-run/examples/blob/main/infinite-scrolling/app/utils/backend.server.ts and use it to supply data for the Link approach that’s outlined in the post.

Infinite scrolling with a `Link` component

The first issue was that after adding app/utils/backend.server.ts to the Remix app, I got this Typescript error:

Element implicitly has an 'any' type because
type 'typeof globalThis' has no index signature.

I’m not sure how far to dig into this. The error is saying that the global object doesn’t have an index signature. From the MDN docs, a function signature defines the input and output of functions or methods. In Typescript an index signature indicates that an object can be accessed using arbitrary keys. That’s what global.__items is trying to do in the backend.server.ts file that I linked to. Typescript expect the property to be declared explicitly.

The trick is to figure out how to declare the property. There are some suggestions in the answers to this Stack Overflow question: https://stackoverflow.com/questions/68481686/type-typeof-globalthis-has-no-index-signature. The most upvoted answer gets close to an approach that works in my Remix app. The only difference was that I had to turn the script into a module with export {};

// types/global.d.ts

declare global {
  var __items: { id: string; value: string }[];
}

export {};

Now back to the actual task 🙂

Probably user-unfriendly for a lot of cases, but this is cool:

// app/routes

import type { LoaderFunctionArgs } from "@remix-run/node";
import { json } from "@remix-run/node";
import { Link, useLoaderData, useLocation } from "@remix-run/react";
import { countItems, getItems } from "~/utils/backend.server";

const LIMIT = 10;

type Item = {
  id: string;
  value: string;
};
type Items = Item[];

export async function loader({ request }: LoaderFunctionArgs) {
  const url = new URL(request.url);
  const page = Number(url.searchParams.get("page")) || 0;
  const start = page * LIMIT;
  const nextPage = page + 1;
  const items = await getItems({ start: start, limit: LIMIT });
  const totalItems = await countItems();

  return json({ items, totalItems, nextPage });
}

export default function LoadMore() {
  const data = useLoaderData<typeof loader>();
  const location = useLocation();
  const newItems: Items = data.items;
  const previousItems: Items = location.state?.items;
  const currentItems: Items = previousItems
    ? previousItems?.concat(newItems)
    : newItems;

  return (
    <div className="max-w-screen-md mx-auto">
      <h1>Load More Data</h1>
      <p>
        Concatenate previous and new items when the "Next Page" button is
        clicked. Items are maintained in the browser's History `state` property.
      </p>
      <div>
        {currentItems.map((item) => (
          <div key={item.id}>{item.value}</div>
        ))}
      </div>
      <Link to={`?page=${data.nextPage}`} state={{ items: currentItems }}>
        Next Page
      </Link>
    </div>
  );
}

The unfriendly part isn’t that it forces users to click a “next page” button, it’s that all the previously loaded data is being pushed to the browser’s History state property. It’s a good proof of concept though.

Here’s a similar approach that uses the React useState hook to maintain state, instead of using location.state:

// app/routes/load-more-use-state.tsx

import type { LoaderFunctionArgs } from "@remix-run/node";
import { json } from "@remix-run/node";
import { useFetcher, Link, useLoaderData, useLocation } from "@remix-run/react";
import { getItems } from "~/utils/backend.server";
import { useEffect, useState } from "react";

const LIMIT = 10;

type Item = {
  id: string;
  value: string;
};
type Items = Item[];
type FetcherData = {
  items: Items;
  nextPage: number;
};

export async function loader({ request }: LoaderFunctionArgs) {
  const url = new URL(request.url);
  const page = Number(url.searchParams.get("page")) || 0;
  const start = page * LIMIT;
  const nextPage = page + 1;
  const items = await getItems({ start: start, limit: LIMIT });

  return json({ items, nextPage });
}

export default function LoadMore() {
  const data = useLoaderData<typeof loader>();
  const fetcher = useFetcher<FetcherData>();
  const [items, setItems] = useState(data.items);
  const [nextPage, setNextPage] = useState(data.nextPage);

  useEffect(() => {
    if (fetcher.data?.nextPage && fetcher.data?.items) {
      setNextPage(fetcher.data?.nextPage);
      const allItems = items.concat(fetcher.data.items);
      setItems(allItems);
    }
  }, [fetcher.data]);

  return (
    <div className="max-w-screen-md mx-auto">
      <h1>Load More Data</h1>
      <p>
        Concatenate previous and new items when the "Next Page" button is
        clicked. New items are concatenated with previous items that have been
        maintained with the React `useState` hook.
      </p>
      <div>
        {items.map((item, i) => (
          <div key={i}>{item.value}</div>
        ))}
      </div>
      <button
        onClick={() => {
          const formData = new FormData();
          formData.append("page", String(nextPage));
          fetcher.submit(formData);
        }}
      >
        Load more posts
      </button>
    </div>
  );
}

That code is using fetcher.submit to make a GET request with its page param set to the nextPage of data. The calls to setItems and setNextPage occur within React useEffect callback function. fetcher.data is in the useEffect dependency array so that the function is only run if fetcher.data has changed.

Infinite scrolling (from first principles(?) and limited knowledge)

(Scroll to the Intersection Observer section at the end of this article to see the approach I ended up using.)

Copying either of the Remix infinite scrolling page examples (https://github.com/remix-run/examples/blob/main/infinite-scrolling/app/routes/page.simple.tsx, https://github.com/remix-run/examples/blob/main/infinite-scrolling/app/routes/page.advanced.tsx) into my app triggers a bunch of errors. I think the issues are that the Remix useTransition hook has been renamed to useNavigation and the react-virtual package has a new maintainer or has been renamed. (I think the go-to package for rendering just a part of a large dataset is now https://www.npmjs.com/package/react-window.)

I want to understand the general pattern that’s needed for infinite scrolling. I’ll work through it step-by-step, starting with the assumption that the component needs to know when a user has scrolled to the bottom of the content’s containing element. When that happens, it will make a call to the server to get more data. The returned data will be appended to the previously loaded data.

Starting with a route that loads data into a scrollable div element:

// app/routes/infinite.tsx

import type { LoaderFunctionArgs } from "@remix-run/node";
import { json } from "@remix-run/node";
import { useLoaderData } from "@remix-run/react";
import { getItems } from "~/utils/backend.server";

const LIMIT = 60;

export async function loader({ request }: LoaderFunctionArgs) {
  const url = new URL(request.url);
  const page = Number(url.searchParams.get("page")) || 0;
  const start = page * LIMIT;
  const nextPage = page + 1;
  const items = await getItems({ start: start, limit: LIMIT });

  return json({ items, nextPage });
}

export default function Infinite() {
  const { items } = useLoaderData<typeof loader>();
  return (
    <div className="max-w-screen-sm mx-auto">
      <h1 className="text-3xl">Infinite</h1>
      <p>
        Automatically load more data when a user scrolls to the bottom of the
        data's containing element.
      </p>
      <div className="overflow-y-scroll max-h-96 mt-6">
        {items.map((item, i) => (
          <div key={i}>{item.value}</div>
        ))}
      </div>
    </div>
  );
}

I’ve updated the backend.server.ts functions so that they mimic network latency:

backend.server.ts

// backend.server.ts

type Item = {
  id: string;
  value: string;
};
export type Items = Item[];

const items = (global.__items =
  global.__items ??
  Array.from({ length: 50_000 }, (_, i) => ({
    id: i.toString(),
    value: `Item ${i}`,
  })));

export async function getItems({
  start,
  limit,
}: {
  start: number;
  limit: number;
}) {
  return new Promise<Items>((resolve) => {
    setTimeout(() => {
      resolve(items.slice(start, start + limit));
    }, 500);
  });
}

export async function getItemsPaginated({
  page,
  limit,
}: {
  page: number;
  limit: number;
}) {
  const start = page * limit;
  return new Promise<Items>((resolve) => {
    setTimeout(() => {
      resolve(items.slice(start, start + limit));
    }, 500);
  });
}

export async function countItems() {
  return new Promise<number>((resolve) => {
    setTimeout(() => {
      resolve(items.length);
    }, 500);
  });
}

Knowing when the content has been scrolled to the bottom

The first step is to figure out how to know when a user has scrolled to here:

This seems like the right place to start:

//...
function handleScroll(event) {}

//...

<div className="overflow-y-scroll max-h-96 mt-6" onScroll={handleScroll}>
{items.map((item, i) => (
  <div key={i}>{item.value}</div>
))}
</div>

//...

With the above change, Typescript gives the warning:

`Parameter ‘event’ implicitly has an ‘any’ type.`

The onScroll event handler has the return type React.UIEventHandler<HTMLDivElement>. It seems that the event handler’s event parameter should be typed (omitting the Handler part of the type):

function handleScroll(event: React.UIEvent<HTMLDivElement>) {}

Details about the scroll event can be found by calling methods on the event.currentTarget property:

event.currentTarget.scrollHeight: returns the height (in pixels) of an element’s content, including content not visible on the screen (due to overflow)
event.currentTarget.scrollTop: gets or sets the number of pixels an element’s content has been scrolled vertically
event.currentTarget.clientHeight: returns the inner height of an element in pixels (includes padding, excludes margins)

Given the above, a user will have scrolled to the end of the available content when the number of pixels they have scrolled vertically, plus the height of the content’s container is equal to the total height of the element’s content:

function handleScroll(event: React.UIEvent<HTMLDivElement>) {
  const currentTarget = event.currentTarget;
  const scrollHeight = currentTarget.scrollHeight;
  const scrollTop = currentTarget.scrollTop;
  const clientHeight = currentTarget.clientHeight;
  const scrolledToBottom = scrollHeight === scrollTop + clientHeight;
  console.log(`scrolledToBottom: ${scrolledToBottom}`);
}

A couple of issues. First how consistent is this across different devices and operating systems? I’m not sure, but I think I should give it a bit of leeway:

const scrolledToBottom = scrollTop + 10 + clientHeight >= scrollHeight;

The other issue is that I think the scroll event needs to be debounced. I haven’t figured out how to access an event inside of a debounce callback, so I’m going to go ahead and use and approach that’s worked in the past.

// using the npm debounce package
import debounce from "debounce";

//...
const debouncedHandleScroll = debounce(
  (scrollHeight: number, scrollTop: number, clientHeight: number) => {
    const scrolledToBottom = scrollTop + 10 + clientHeight >= scrollHeight;
    console.log(scrolledToBottom);
  },
  200
);

function handleScroll(event: React.UIEvent<HTMLDivElement>) {
  const currentTarget = event.currentTarget;
  const scrollHeight = currentTarget.scrollHeight;
  const scrollTop = currentTarget.scrollTop;
  const clientHeight = currentTarget.clientHeight;
  debouncedHandleScroll(scrollHeight, scrollTop, clientHeight);
}

Loading more content

When scrolledToBottom === true the debouncedHandleScroll function will increment the page ref by one and make a call to fetcher.load to get the next page of data. (Edit: I used a useRef instead of useState to track the page, but thinking about it some more, it would probably be ok to use state here):

//...
import { useRef } from "react";
//...

const page = useRef(0);

const debouncedHandleScroll = debounce(
  (scrollHeight: number, scrollTop: number, clientHeight: number) => {
    const scrolledToBottom = scrollTop + 10 + clientHeight >= scrollHeight;
    if (scrolledToBottom) {
      page.current += 1;
      fetcher.load(`/infinite?page=${page.current}`)
    }
  },
  200
);

So far so good:

useState seems like the right approach for keeping track of the components items though. At the risk of blowing up my machine:

//...
import { useRef, useState } from "react";

//...
/**
 * imported from backend.server.ts
 * type Item = {id: string; value: string;};
 */
type FetcherData = {
  items: Items;
};

//...
export default function Infinite() {
  const data = useLoaderData<typeof loader>();
  const fetcher = useFetcher<FetcherData>();
  const [items, setItems] = useState(data.items);
  const page = useRef(0);

  if (fetcher?.data && fetcher.data?.items) {
    const allItems = items.concat(fetcher.data.items);
    setItems(allItems);
  }

It could have been worse 🙂

I think this is an appropriate use case for the useEffect hook. The description in the React docs describes this exact use case: “a React Hook that lets you synchronize a component with an external system.” Code in a useEffect callback function isn’t run until after the component has rendered. The error I’m getting is being caused by calling setItems while the component is rendering.

useEffect also accepts an array of dependencies. If a dependency is supplied, the callback function will only be run if the dependency’s value has changed since the previous render.

That’s better. I’ve also added a fetcher.state === "idle" condition to the debouncedHandleScroll function. That will prevent fetcher.load from being called when the fetcher is in its "loading" state:

//...
import { useEffect, useRef, useState } from "react";

//...

export default function Infinite() {
  const data = useLoaderData<typeof loader>();
  const fetcher = useFetcher<FetcherData>();
  const [items, setItems] = useState(data.items);
  const page = useRef(0);

  useEffect(() => {
    if (fetcher?.data && fetcher.data?.items) {
      const allItems = items.concat(fetcher.data.items);
      setItems(allItems);
    }
  }, [fetcher.data?.items]);

  const debouncedHandleScroll = debounce(
    (scrollHeight: number, scrollTop: number, clientHeight: number) => {
      const scrolledToBottom = scrollTop + 10 + clientHeight >= scrollHeight;
      if (scrolledToBottom && fetcher.state === "idle") {
        page.current += 1;
        fetcher.load(`/infinite?page=${page.current}`);
      }
    },
    200
  );

It works!

Todo:

Minor issues:

the UI should give a visual indication that content is loading
when new content is loaded, the movement of the scroll bar’s slider is kind of janky

Major(?) issues:

The data isn’t being pruned. Previously scrolled items that are outside of the viewable element are being held in memory on the client.
The data isn’t being windowed. If 1000 items have been sent to the client, React has to deal with all items each time new data is received. I don’t think it’s technically correct to say that React will re-render all the items, but it will have to maintain them in the VDOM. I’m assuming that the reconciliation process of comparing the new VDOM to the previous VDOM is less efficient when the VDOM is larger. Related to this, the Infinite component is taking a bit of a risk when it maps the items with items.map((item, i), then uses i as the value of the key for the rendered item. It should be using item.id instead of the map callback’s index to set the key. The key helps React keep track of what items have changed. It’s only by luck the the index and item.id are identical in the current version of the route. Possibly if the data was “pruned,” it wouldn’t need to be “windowed,” otherwise I should look at something like react-window for windowing.
There isn’t a way to share a link to a particular place in the list of items, for example to share a link that would start at item.id 100.
If there was a way to share a link that started at a particular place in the list, there isn’t a way to load data in reverse. For example, if /infinite/100 opened the list with at the 100th item, there isn’t a way to scroll backwards to see item 99, 98, 97…

Enter a list from a given page

My initial thought for allowing a list to be loaded starting from a point other than page=0 was to load the list starting from a specific item.id. Thinking about it some more, that could add some complexity to the app that I’d rather not deal with yet. The problem is that the code would need to ensure that duplicate list items weren’t loaded. To avoid that, it would need to keep track of the initial item.id, then use that id in all pagination requests. To get around that, instead of loading the list from a specific item.id, I’ll allow the list to be loaded from the page that contains the requested item.id.

To simplify things, I’ve created an alternate version of the backend.server.ts file that stores 500 items in a local variable. Having 50 000 items in a global variable seemed to be causing some unexpected issues:

const localItems = Array.from({ length: 500 }, (_, i) => ({
  id: i.toString(),
  value: `Item ${i}`,
}));

I’ve also added a getIndexFor function that returns the index of a given item id:

export async function getIndexFor({ id }: { id: string }) {
  return new Promise<number>((resolve) => {
    setTimeout(() => {
      let index = localItems.findIndex((item) => {
        let result = item.id === id;
        if (result) {
        }
        return result;
      });

      resolve(index);
    }, 500);
  });
}

backednAlt.server.ts

// utils/backend.Alt.server.ts

export type Item = {
  id: string;
  value: string;
};
export type Items = Item[];

const localItems = Array.from({ length: 500 }, (_, i) => ({
  id: i.toString(),
  value: `Item ${i}`,
}));

export async function getItems({
  start,
  limit,
}: {
  start: number;
  limit: number;
}) {
  return new Promise<Items>((resolve) => {
    setTimeout(() => {
      resolve(localItems.slice(start, start + limit));
    }, 500);
  });
}

export async function getItemsFrom({
  after,
  limit,
}: {
  after: string;
  limit: number;
}) {
  return new Promise<Items>((resolve) => {
    setTimeout(() => {
      let index = localItems.findIndex((i) => i.id === after);
      resolve(localItems.slice(index, index + limit));
    }, 500);
  });
}

export async function getIndexFor({ id }: { id: string }) {
  return new Promise<number>((resolve) => {
    setTimeout(() => {
      let index = localItems.findIndex((item) => {
        let result = item.id === id;
        if (result) {
        }
        return result;
      });

      resolve(index);
    }, 500);
  });
}

export async function getItemsPaginated({
  page,
  limit,
}: {
  page: number;
  limit: number;
}) {
  const start = page * limit;
  return new Promise<Items>((resolve) => {
    setTimeout(() => {
      resolve(localItems.slice(start, start + limit));
    }, 500);
  });
}

export async function countItems() {
  return new Promise<number>((resolve) => {
    setTimeout(() => {
      resolve(localItems.length);
    }, 500);
  });
}

I’m (re-)implementing this as I write. The code is as basic as I can make it. The goal is to understand what’s going on.

Set page to either the value of the page query param or 0. Set lastSeenId to either the value of the lastSeenId query param or null. If lastSeenId is not null and page === 0 (a page query param was not set), find the page (chunk) from the items array that contains the lastSeenId. Set it as the value of page:

// app/routes/start-from-page.tsx

//...
const LIMIT = 5;

export async function loader({ request }: LoaderFunctionArgs) {
  const url = new URL(request.url);
  let page = Number(url.searchParams.get("page")) || 0;
  const lastSeenId = url.searchParams?.get("lastSeenId");

  if (lastSeenId && page === 0) {
    const lastSeenIndex = await getIndexFor({ id: lastSeenId });
    page = Math.floor(lastSeenIndex / LIMIT);
  }

  const start = page * LIMIT;
  const items = await getItems({ start, limit: LIMIT });

  return json({ items, initialPage: page });
}

export default function StartFromPage() {
  const data = useLoaderData<typeof loader>();
  console.log(JSON.stringify(data.items, null, 2));

  return (
    <div>
      <h1>Start From Page</h1>
    </div>
  );
}

Visiting http://localhost:5173/start-from-page:

// initialPage:
0
// items:
[
  {
    "id": "0",
    "value": "Item 0"
  },
  {
    "id": "1",
    "value": "Item 1"
  },
  {
    "id": "2",
    "value": "Item 2"
  },
  {
    "id": "3",
    "value": "Item 3"
  },
  {
    "id": "4",
    "value": "Item 4"
  }
]

http://localhost:5173/start-from-page?page=1: initialPage is set to 1, items starting from item.id: "5" are returned.

http://localhost:5173/start-from-page?lastSeenId=11: initialPage is set to 2, items starting from item.id: "10" are returned.

Add a CopyButton component

// app/components/CopyButton.tsx

import { useState } from "react";

interface CopyButtonProps {
  url: string;
}

export default function CopyButton({ url }: CopyButtonProps) {
  const [isCopied, setIsCopied] = useState(false);

  const handleClick = async () => {
    try {
      await navigator.clipboard.writeText(url);
      setIsCopied(true);
      setTimeout(() => setIsCopied(false), 2000);
    } catch (err) {
      console.error("Error copying link to clipboard");
    }
  };

  return (
    <div className="flex w-full justify-end">
      <button
        className="border-slate-700 border rounded px-2 py-1"
        onClick={handleClick}
      >
        {isCopied ? "Copied" : "Copy Link"}
      </button>
    </div>
  );
}

This is a quick attempt to mimic something like a “share” link that would be used in a comments list. Clicking the button copies a URL to the browser’s clipboard.

Added to the route’s component, clicking an item’s “Copy” button will add a URL that has a (poorly named, but I’ll go with it for now) lastSeenId query parameter:

// app/routes/start-from-page.tsx

//...
return (
  <div className="max-w-screem-sm mx-auto">
    <h1>Start From Page</h1>
    <div className="relative">
      <div
        className="overflow-y-scroll max-h-96 mt-6 divide-y divide-slate-300"
        onScroll={handleScroll}
      >
        {currentItems?.map((item) => (
          <div key={item.id} className="px-3 py-6">
            <p>{item.value}</p>
            <CopyButton
              url={`http://localhost:5173/start-from-page?lastSeenId=${item.id}`}
            />
          </div>
        ))}
      </div>
    </div>
  </div>
);
//...

Handle scrolling to the bottom of the items list

If the viewable area has been scrolled to the bottom, set loadingDirRef to "forward". Increment pageRef.current by 1. Calculate the value of the next page by adding pageRef.current to the value of the initial page that was sent from the loader when the route was first entered.

// app/routes/start-from-page.tsx

export default function StartFromPage() {
  const { items, initialPage } = useLoaderData<typeof loader>();
  const fetcher = useFetcher<FetcherData>();
  const [currentItems, setCurrentItems] = useState(items);
  const pageRef = useRef(0);
  const loadingDirRef = useRef<LoadingDirection>("forward");

  useEffect(() => {
    if (fetcher?.data && fetcher?.data?.items) {
      setCurrentItems(
        loadingDirRef.current === "forward"
          ? currentItems.concat(fetcher.data.items)
          : []
      );
    }
  }, [fetcher.data?.items]);

  const debouncedHandleScroll = debounce(
    (scrollHeight: number, scrollTop: number, clientHeight: number) => {
      const scrolledToBottom = scrollTop + 10 + clientHeight >= scrollHeight;
      if (scrolledToBottom && fetcher.state === "idle") {
        loadingDirRef.current = "forward";
        pageRef.current += 1;
        const nextPage = initialPage + pageRef.current;
        fetcher.load(`/start-from-page?page=${nextPage}`);
      }
    }
  );

  function handleScroll(event: React.UIEvent<HTMLDivElement>) {
    const currentTarget = event.currentTarget;
    const scrollHeight = currentTarget.scrollHeight;
    const scrollTop = currentTarget.scrollTop;
    const clientHeight = currentTarget.clientHeight;
    debouncedHandleScroll(scrollHeight, scrollTop, clientHeight);
  }

Handle scrolling to the top of the currently loaded list of items

This is where things get tricky. The code following code will be as explicit as possible.

Instead of tracking the page with a single pageRef, use forwardPageRef to track forward scrolling, and backwardPageRef to track backward scrolling:

const forwardPageRef = useRef(0);
const backwardPageRef = useRef(0);

The viewable content is being scrolled backward(s) if scrollTop === 0. For this case, make sure the previousPage value is greater than 0 before decrementing backwardPageRef.current and making the request to get the previous page data. (I’m trying to think of a case where not updating previousPageRef.current would be strictly necessary. What really matters is not making the request if initialPage + previousPageRef.current is less than 0)

const debouncedHandleScroll = debounce(
  (scrollHeight: number, scrollTop: number, clientHeight: number) => {
    const scrolledToBottom = scrollTop + 10 + clientHeight >= scrollHeight;
    if (scrolledToBottom && fetcher.state === "idle") {
      loadingDirRef.current = "forward";
      forwardPageRef.current += 1;
      const nextPage = initialPage
        ? initialPage + forwardPageRef.current
        : forwardPageRef.current;
      fetcher.load(`/start-from-page?page=${nextPage}`);
    }
    if (scrollTop === 0 && fetcher.state === "idle") {
      loadingDirRef.current = "backward";
      const tryPageFromRef = backwardPageRef.current - 1;
      const previousPage = initialPage
        ? initialPage + tryPageFromRef
        : tryPageFromRef;
      if (previousPage >= 0) {
        backwardPageRef.current -= 1;
        fetcher.load(`/start-from-page?page=${previousPage}`);
      }
    }
  }

Then handle both forward and backward pagination if fetcher.data.items has been updated:

useEffect(() => {
  if (fetcher?.data && fetcher.data?.items) {
    setCurrentItems(
      loadingDirRef.current === "forward"
        ? currentItems.concat(fetcher.data.items)
        : fetcher.data.items.concat(currentItems)
    );
  }
}, [fetcher.data?.items]);

The last thing that needs to be taken care of is that the loader function needs to handle the following cases:

the lastSeenId query param is set and the page query param is not set
the page query param is set and the lastSeenId param is not set
both the page and lastSeenId params are set

export async function loader({ request }: LoaderFunctionArgs) {
  const url = new URL(request.url);
  const pageParam = url.searchParams.get("page");
  const lastSeenId = url.searchParams.get("lastSeenId");

  let page;

  if (pageParam !== null) {
    const parsedPage = Number(pageParam);
    page = !isNaN(parsedPage) ? parsedPage : 0;
  }

  if (lastSeenId !== null && page === undefined) {
    const lastSeenIndex = await getIndexFor({ id: lastSeenId });
    if (lastSeenIndex >= 0) {
      page = Math.floor(lastSeenIndex / LIMIT);
    } else {
      page = 0;
    }
  }

  page = page !== undefined ? page : 0;

  const start = page * LIMIT;
  const items = await getItems({ start, limit: LIMIT });

  return json({ items, initialPage: page });
}

For (my own) reference, here’s the full route file. It works as expected, with the notable issue of scrollTop not being properly handled when new data is prepended to existing data. I’ll ignore that issue for now.

app/routes/start-from-page.tsx

// app/routes/start-from-page.tsx

import type { LoaderFunctionArgs } from "@remix-run/node";
import { json } from "@remix-run/node";
import { useFetcher, useLoaderData } from "@remix-run/react";
import debounce from "debounce";

import { useEffect, useRef, useState } from "react";

import type { Items } from "~/utils/backendAlt.server";
import { getItems, getIndexFor } from "~/utils/backendAlt.server";
import CopyButton from "~/components/CopyButton";

const LIMIT = 10;

type FetcherData = {
  items: Items;
};
type LoadingDirection = "forward" | "backward";

export async function loader({ request }: LoaderFunctionArgs) {
  const url = new URL(request.url);
  const pageParam = url.searchParams.get("page");
  const lastSeenId = url.searchParams.get("lastSeenId");

  let page;

  if (pageParam !== null) {
    const parsedPage = Number(pageParam);
    page = !isNaN(parsedPage) ? parsedPage : 0;
  }

  if (lastSeenId !== null && page === undefined) {
    const lastSeenIndex = await getIndexFor({ id: lastSeenId });
    if (lastSeenIndex >= 0) {
      page = Math.floor(lastSeenIndex / LIMIT);
    } else {
      page = 0;
    }
  }

  page = page !== undefined ? page : 0;

  const start = page * LIMIT;
  const items = await getItems({ start, limit: LIMIT });

  return json({ items, initialPage: page });
}

export default function StartFromPage() {
  const { items, initialPage } = useLoaderData<typeof loader>();
  const fetcher = useFetcher<FetcherData>();
  const [currentItems, setCurrentItems] = useState(items);
  const forwardPageRef = useRef(0);
  const backwardPageRef = useRef(0);
  const loadingDirRef = useRef<LoadingDirection>("forward");

  useEffect(() => {
    if (fetcher?.data && fetcher.data?.items) {
      setCurrentItems(
        loadingDirRef.current === "forward"
          ? currentItems.concat(fetcher.data.items)
          : fetcher.data.items.concat(currentItems)
      );
    }
  }, [fetcher.data?.items]);

  const debouncedHandleScroll = debounce(
    (scrollHeight: number, scrollTop: number, clientHeight: number) => {
      const scrolledToBottom = scrollTop + 10 + clientHeight >= scrollHeight;
      if (scrolledToBottom && fetcher.state === "idle") {
        loadingDirRef.current = "forward";
        forwardPageRef.current += 1;
        const nextPage = initialPage
          ? initialPage + forwardPageRef.current
          : forwardPageRef.current;
        fetcher.load(`/start-from-page?page=${nextPage}`);
      }
      if (scrollTop === 0 && fetcher.state === "idle") {
        loadingDirRef.current = "backward";
        const tryPageFromRef = backwardPageRef.current - 1;
        const previousPage = initialPage
          ? initialPage + tryPageFromRef
          : tryPageFromRef;
        if (previousPage >= 0) {
          backwardPageRef.current -= 1;
          fetcher.load(`/start-from-page?page=${previousPage}`);
        }
      }
    }
  );

  function handleScroll(event: React.UIEvent<HTMLDivElement>) {
    const currentTarget = event.currentTarget;
    const scrollHeight = currentTarget.scrollHeight;
    const scrollTop = currentTarget.scrollTop;
    const clientHeight = currentTarget.clientHeight;
    debouncedHandleScroll(scrollHeight, scrollTop, clientHeight);
  }

  return (
    <div className="max-w-screem-sm mx-auto">
      <h1>Start From Page</h1>
      <div className="relative">
        <div
          className="overflow-y-scroll max-h-96 mt-6 divide-y divide-slate-300"
          onScroll={handleScroll}
        >
          {currentItems?.map((item) => (
            <div key={item.id} className="px-3 py-6">
              <p>{item.value}</p>
              <CopyButton
                url={`http://localhost:5173/start-from-page?lastSeenId=${item.id}`}
              />
            </div>
          ))}
        </div>
      </div>
    </div>
  );
}

Pruning data

I’m tempted to just post this without comment:

app/routes/infinite-prune.tsx

// app/routes/infinite-prune.tsx

import type { LoaderFunctionArgs } from "@remix-run/node";
import { json } from "@remix-run/node";
import { useFetcher, useLoaderData } from "@remix-run/react";
import debounce from "debounce";

import { useEffect, useRef, useState } from "react";

import type { Items } from "~/utils/backendAlt.server";
import { getItems, getIndexFor } from "~/utils/backendAlt.server";

const LIMIT = 50;

type PagedItems = {
  [page: string]: Items;
};

type FetcherData = {
  pagedItems: PagedItems;
};

type LoadingDirection = "forward" | "backward";

export async function loader({ request }: LoaderFunctionArgs) {
  const url = new URL(request.url);
  const pageParam = url.searchParams.get("page");
  const lastSeenId = url.searchParams.get("lastSeenId");

  let page;

  if (pageParam !== null) {
    const parsedPage = Number(pageParam);
    page = !isNaN(parsedPage) ? parsedPage : 0;
  }

  if (lastSeenId !== null && page === undefined) {
    const lastSeenIndex = await getIndexFor({ id: lastSeenId });
    if (lastSeenIndex >= 0) {
      page = Math.floor(lastSeenIndex / LIMIT);
    } else {
      page = 0;
    }
  }

  page = page !== undefined ? page : 0;

  const start = page * LIMIT;
  const items = await getItems({ start, limit: LIMIT });
  const pagedItems = {
    [page]: items,
  };

  return json({ pagedItems, initialPage: page });
}

export default function InfinitePrune() {
  const { pagedItems, initialPage } = useLoaderData<typeof loader>();
  const fetcher = useFetcher<FetcherData>();
  const [pages, setPages] = useState(pagedItems);
  const forwardPageRef = useRef(0); // tracks how many pages forward have been requested
  const backwardPageRef = useRef(0); // tracks how many pages backward have been requested
  const loadingDirRef = useRef<LoadingDirection>("forward"); // "forward" or "backward", needed to adjust forward/backward pageRefs in prune function

  useEffect(() => {
    if (fetcher?.data && fetcher.data?.pagedItems) {
      let summedPages = { ...pages, ...fetcher.data.pagedItems };
      summedPages = prunePages(summedPages);
      setPages(summedPages);
    }
  }, [fetcher.data?.pagedItems]);

  const prunePages = (pages: PagedItems) => {
    let pruned;
    if (Object.keys(pages).length > 5) {
      if (loadingDirRef.current === "forward") {
        pruned = removeFirstPage(pages);
        backwardPageRef.current += 1;
      } else {
        pruned = removeLastPage(pages);
        forwardPageRef.current -= 1;
      }
    }

    return pruned ? pruned : pages;
  };

  function removeFirstPage(pages: PagedItems) {
    const [firstKey, ...restKeys] = Object.keys(pages);
    const { [firstKey]: _, ...prunedPages } = pages; // _ is a discard variable for the first property
    return prunedPages;
  }

  function removeLastPage(pages: PagedItems) {
    const keys = Object.keys(pages);
    const lastKey = keys[keys.length - 1];
    const { [lastKey]: _, ...prunedPages } = pages; // Exclude the last property
    return prunedPages;
  }

  const debouncedHandleScroll = debounce(
    (scrollHeight: number, scrollTop: number, clientHeight: number) => {
      const scrolledToBottom = scrollTop + 10 + clientHeight >= scrollHeight;
      if (scrolledToBottom && fetcher.state === "idle") {
        loadingDirRef.current = "forward";
        forwardPageRef.current += 1;
        const nextPage = initialPage
          ? initialPage + forwardPageRef.current
          : forwardPageRef.current;
        fetcher.load(`/infinite-prune?page=${nextPage}`);
      }
      if (scrollTop === 0 && fetcher.state === "idle") {
        const tryPageFromRef = backwardPageRef.current - 1;
        const previousPage = initialPage
          ? initialPage + tryPageFromRef
          : tryPageFromRef;
        if (previousPage >= 0) {
          loadingDirRef.current = "backward";
          backwardPageRef.current -= 1;
          fetcher.load(`/infinite-prune?page=${previousPage}`);
        }
      }
    }
  );

  function handleScroll(event: React.UIEvent<HTMLDivElement>) {
    const currentTarget = event.currentTarget;
    const scrollHeight = currentTarget.scrollHeight;
    const scrollTop = currentTarget.scrollTop;
    const clientHeight = currentTarget.clientHeight;
    debouncedHandleScroll(scrollHeight, scrollTop, clientHeight);
  }

  return (
    <div className="max-w-screen-sm mx-auto">
      <h1>Infinite Prune</h1>
      <div className="relative">
        <div
          className="overflow-y-scroll max-h-96 mt-6 divide-y divide-slate-300"
          onScroll={handleScroll}
        >
          {Object.entries(pages).map(([page, items]) => (
            <div key={page}>
              {items.map((item) => (
                <div key={item.id} className="px-3 py-6">
                  <p>{item.value}</p>
                </div>
              ))}
            </div>
          ))}
        </div>
      </div>
    </div>
  );
}

The main difference from the past couple of infinite scrolling implementations is that it passes data from the loader function as an object, with the page number as the object’s key:

//...  
  const items = await getItems({ start, limit: LIMIT });
  const pagedItems = {
    [page]: items,
  };

  return json({ pagedItems, initialPage: page });
}

This seems essential for pruning the data by page number, but it also simplifies adding pages to the already fetched data:

export default function InfinitePrune() {
  const { pagedItems, initialPage } = useLoaderData<typeof loader>();
  const fetcher = useFetcher<FetcherData>();
  const [pages, setPages] = useState(pagedItems);
  const forwardPageRef = useRef(0); // tracks how many pages forward have been requested
  const backwardPageRef = useRef(0); // tracks how many pages backward have been requested
  const loadingDirRef = useRef<LoadingDirection>("forward"); // "forward" or "backward", needed to adjust forward/backward pageRefs in prune function

  useEffect(() => {
    if (fetcher?.data && fetcher.data?.pagedItems) {
      let summedPages = { ...pages, ...fetcher.data.pagedItems };
      summedPages = prunePages(summedPages);
      setPages(summedPages);
    }
  }, [fetcher.data?.pagedItems]);

The pruning function is basic, just a proof of concept:

const prunePages = (pages: PagedItems) => {
  let pruned;
  if (Object.keys(pages).length > 5) {
    if (loadingDirRef.current === "forward") {
      pruned = removeFirstPage(pages);
      backwardPageRef.current += 1;
    } else {
      pruned = removeLastPage(pages);
      forwardPageRef.current -= 1;
    }
  }

  return pruned ? pruned : pages;
};

function removeFirstPage(pages: PagedItems) {
  const [firstKey, ...restKeys] = Object.keys(pages);
  const { [firstKey]: _, ...prunedPages } = pages; // _ is a discard variable for the first property
  return prunedPages;
}

function removeLastPage(pages: PagedItems) {
  const keys = Object.keys(pages);
  const lastKey = keys[keys.length - 1];
  const { [lastKey]: _, ...prunedPages } = pages; // Exclude the last property
  return prunedPages;
}

I won’t be adding this to a production app until there’s a real need for it. It seems likely there’s a trade off between removing data from the client, and forcing the client to make network requests to retrieve data that was removed. I also need to be certain that the prunePages function is actually freeing up memory on the client. I’ll investigate more if the production app (a comment system) starts getting huge numbers of comments.

Windowing data

Windowing, in the context of a React application means to only render the data that’s currently viewable. In the examples I’ve posted above, clientHeight is the viewable area and scrollHeight is the total height of the content. As data is pulled to the client with calls to fetcher.load, most of the data is outside of the viewable area. Each time data is loaded, the component’s state is updated with something like:

useEffect(() => {
  if (fetcher?.data && fetcher.data?.pagedItems) {
    let summedPages = { ...pages, ...fetcher.data.pagedItems };
    summedPages = prunePages(summedPages);
    setPages(summedPages);
  }
}, [fetcher.data?.pagedItems]);

Updating the state causes the component to re-render. The more items there are to be re-rendered, the more memory is used by the process.

That’s my super basic understanding of windowing. What I don’t know (yet) is at what point it becomes an issue. For a comment system, would rendering 100 comments be a problem? I’m taking a guess that rendering 1000 comments would be a problem. For the comment system I’m developing, I want to at least make sure that it’s structured in a way that will allow windowing to be added at a later time.

The go to library for windowing seems to be react-window:

$ npm install --save react-window
$ npm install --save @types/react-window

React Window FixedSizeList

Here’s a basic Remix route that uses react-window’s FixedSizeList component:

// app/routes/fixed-size-list-basic.tsx

import { FixedSizeList as List } from "react-window";

const LIMIT = 100;

const Row = ({
  index,
  style,
}: {
  index: number;
  style: React.CSSProperties;
}) => <div style={style}>Row {index}</div>;

export default function FixedSizeListBasic() {
  return (
    <div className="max-w-screen-sm mx-auto">
      <h1>React Window</h1>
      <List
        className="my-8"
        height={400}
        itemCount={LIMIT}
        itemSize={70}
        width={600}
      >
        {Row}
      </List>
    </div>
  );
}

Scrolling the list with the React dev tools “Highlight updates when components rendered” option enabled, I can see that only the list items that are within the viewable area, plus a few items at the top and bottom of the viewable area are rendered when the list is scrolled. (The number of top and bottom items outside of the viewable area that are rendered can be set with the overscanCount prop. It defaults to 2?)

(There’s a fairly obvious trade off in terms of efficiency here. In the previous examples in this post, components were only re-rendered when new data was pulled in. With React Window, fewer items are initially rendered, but they are being rendered on each scroll event. I’ll ignore this for now.)

In the example above, the Row function is being passed to the List component. The List component expects to be passed a function that can receive index and style props. The function can also be sent a data prop. It is passed to the function from the List component using the list component’s itemData prop.

Here’s an example with an Items function that’s passed data from the loader. I had trouble getting the List component’s itemCount value to be updated dynamically. The approach used below works. The trick is to make sure that the value passed to the itemCount prop matches the current state:

import type { LoaderFunctionArgs } from "@remix-run/node";
import { json } from "@remix-run/node";
import { useFetcher, useLoaderData } from "@remix-run/react";
import { useEffect, useRef, useState } from "react";

import { FixedSizeList as List } from "react-window";

import { getItems } from "~/utils/backendAlt.server";
import type { Items } from "~/utils/backendAlt.server";

const LIMIT = 10;

type PagedItems = {
  [page: string]: Items;
};

type FetcherData = {
  pagedItems: PagedItems;
};

export async function loader({ request }: LoaderFunctionArgs) {
  const url = new URL(request.url);
  const page = Number(url.searchParams.get("page")) || 0;
  const start = page * LIMIT;
  const items = await getItems({ start, limit: LIMIT });

  const pagedItems = {
    [page]: items,
  };

  return json({ pagedItems });
}

const Item = ({
  index,
  style,
  data,
}: {
  index: number;
  style: React.CSSProperties;
  data: Items;
}) => <div style={style}>{data[index].value}</div>;

export default function FixedSizeListBasic() {
  const { pagedItems } = useLoaderData<typeof loader>();
  const fetcher = useFetcher<FetcherData>();
  const pageRef = useRef(Number(Object.keys(pagedItems)[0]));
  const [pages, setPages] = useState(pagedItems);
  const flattenedPagesRef = useRef(Object.values(pages).flat(1)); // not required here, but calling `flat(1)` will prevent nexted arrays from  being flattened
  const [itemCount, setItemCount] = useState(flattenedPagesRef.current.length);

  useEffect(() => {
    flattenedPagesRef.current = Object.values(pages).flat(1);
    setItemCount(flattenedPagesRef.current.length);
  }, [pages]);

  useEffect(() => {
    if (fetcher?.data && fetcher.data?.pagedItems) {
      let summedPages = { ...pages, ...fetcher.data.pagedItems };
      setPages(summedPages);
    }
  }, [fetcher?.data?.pagedItems]);

  function handleSubmit() {
    if (fetcher.state === "idle") {
      pageRef.current += 1;
      fetcher.load(`/fixed-size-list-basic?page=${pageRef.current}`);
    }
  }

  return (
    <div className="max-w-screen-sm mx-auto">
      <h1>React Window</h1>
      <List
        className="my-8"
        height={400}
        itemCount={itemCount}
        itemSize={70}
        width={600}
        itemData={flattenedPagesRef.current}
      >
        {Item}
      </List>
      <button type="submit" onClick={handleSubmit}>
        Load More
      </button>
    </div>
  );
}

React Window VariableSizeList

VariableShttps://react-window.vercel.app/#/api/VariableSizeListizeList: like FixSizeList but for displaying list items with a variable size.

To test it out, I’ve added a (crude) buildItem function to backendIpsum.server.ts:

const text =
  "Raskolnikov was already entering the room. 
He came in looking as though he had the 
utmost difficulty not to burst out laughing 
again. Behind him Razumihin strode in gawky and
 awkward, shamefaced and red as a peony, 
with an utterly crestfallen and ferocious 
expression. His face and whole figure...";

const sentences = text.split(".");

function selectSentence(sentences: string[]) {
  const index = Math.floor(Math.random() * sentences.length);
  return sentences[index];
}

function buildItem() {
  const itemLength = Math.floor(Math.random() * 30);
  let result = [];
  for (let i = 0; i < itemLength; i++) {
    result.push(selectSentence(sentences));
  }
  return result.join(". ").trim();
}

const localItems = Array.from({ length: 500 }, (_, i) => ({
  id: i.toString(),
  value: `Entry ${i}: ${buildItem()}`,
}));

Nice 🙂

The screenshot above indicates what can go wrong when attempting to use a VariableSizeList component with user generated content. I don’t think this use case is fully supported by React Window, but it seems that there are some workarounds. I’ve tried a few.

Here’s the best I’ve come up with. It makes some assumptions that are unlikely to be true:

// app/routes/window-variable.tsx

import type { LoaderFunctionArgs } from "@remix-run/node";
import { json } from "@remix-run/node";
import { useFetcher, useLoaderData } from "@remix-run/react";
import { useCallback, useEffect, useRef, useState } from "react";

import { VariableSizeList as List, VariableSizeList } from "react-window";

import { getItems } from "~/utils/backendIpsum.server";
import type { Items } from "~/utils/backendIpsum.server";

const LIMIT = 10;

type PagedItems = {
  [page: string]: Items;
};

type FetcherData = {
  pagedItems: PagedItems;
};

export async function loader({ request }: LoaderFunctionArgs) {
  const url = new URL(request.url);
  const page = Number(url.searchParams.get("page")) || 0;
  const start = page * LIMIT;
  const items = await getItems({ start, limit: LIMIT });

  const pagedItems = {
    [page]: items,
  };

  return json({ pagedItems });
}

export default function VariableSizeListTest() {
  const { pagedItems } = useLoaderData<typeof loader>();
  const fetcher = useFetcher<FetcherData>();
  const pageRef = useRef(Number(Object.keys(pagedItems)[0]));
  const [pages, setPages] = useState(pagedItems);
  const flattenedPagesRef = useRef(Object.values(pages).flat(1)); // not required here, but calling `flat(1)` will prevent nexted arrays from  being flattened
  const [itemCount, setItemCount] = useState(flattenedPagesRef.current.length);

  function handleSubmit() {
    if (fetcher.state === "idle") {
      pageRef.current += 1;
      fetcher.load(`/window-variable?page=${pageRef.current}`);
    }
  }

  useEffect(() => {
    flattenedPagesRef.current = Object.values(pages).flat(1);
    setItemCount(flattenedPagesRef.current.length);
  }, [pages]);

  useEffect(() => {
    if (fetcher?.data && fetcher.data?.pagedItems) {
      let summedPages = { ...pages, ...fetcher.data.pagedItems };
      setPages(summedPages);

      const newHeights = [...itemHeights];
      // This could be optimized a bit.
      const additionalItems = Object.values(fetcher.data.pagedItems).flat();
      additionalItems.forEach(() => newHeights.push(200)); // 200 is being set as the default height
      setItemHeights(newHeights);
    }
  }, [fetcher?.data?.pagedItems]);

  const listRef = useRef<VariableSizeList>(null);
  const [itemHeights, setItemHeights] = useState(Array(itemCount).fill(200));
  const getRowHeight = (index: number) => {
    const itemHeight = itemHeights[index];
    console.log(`itemHeight: ${itemHeight}`);
    return itemHeight;
  };

  // A proof of concept to show that heights can be adjusted. Don't use this!
  function estimateHeight(text: string, charsPerLine = 40, lineHeight = 20) {
    const lines = Math.ceil(text.length / charsPerLine);
    return lines * lineHeight;
  }

  const measureRow = useCallback(
    (node: HTMLDivElement | null, index: number) => {
      if (node !== null) {
        const text = node.innerText;
        const height = estimateHeight(text);
        // this will always return 200.
        //  const height = node.getBoundingClientRect().height;
        if (itemHeights[index] !== height) {
          const newHeights = [...itemHeights];
          newHeights[index] = height;
          setItemHeights(newHeights);
          if (listRef.current) {
            listRef.current.resetAfterIndex(index);
          }
        }
      }
    },
    [itemHeights]
  );

  const Item = ({
    index,
    style,
    data,
  }: {
    index: number;
    style: React.CSSProperties;
    data: Items;
  }) => {
    return (
      <div ref={(node) => measureRow(node, index)} style={style}>
        {data[index].value}
      </div>
    );
  };

  return (
    <div className="max-w-screen-sm mx-auto">
      <List
        ref={listRef}
        height={800}
        itemCount={itemCount}
        itemSize={getRowHeight}
        estimatedItemSize={200}
        width={600}
        itemData={flattenedPagesRef.current}
      >
        {Item}
      </List>

      <button
        type="submit"
        onClick={handleSubmit}
        disabled={fetcher.state !== "idle"}
      >
        Load More
      </button>
    </div>
  );
}

I’ll leave it there 🙂

Infinite scrolling with the Intersection Observer API

The last section of this article suggests lazy loading elements with the Intersection Observer API: https://www.uber.com/en-IN/blog/supercharge-the-way-you-render-large-lists-in-react/.

The Intersection Observer API provides a way to asynchronously observe changes in the intersection of a target element with an ancestor element or with a top-level document’s viewport.
MDN docs

IntersectionObserver has good browser support: https://caniuse.com/intersectionobserver.

There’s a React Intersection Observer package that’s actively maintained and well documented: https://www.npmjs.com/package/react-intersection-observer.

This article gives a good overview of how to implement infinite scrolling in a React app with React Intersection Observer: https://medium.com/@nikitaprus98/infinite-scroll-in-reactjs-6a38d0ebe8af.

Putting that together, here’s a basic implementation in a Remix route:

// app/routes/infinite-intersection-observer.tsx

import type { LoaderFunctionArgs } from "@remix-run/node";
import { json } from "@remix-run/node";
import { useFetcher, useLoaderData } from "@remix-run/react";
import { useEffect, useMemo, useRef, useState } from "react";
import { useInView } from "react-intersection-observer";

import type { Items } from "~/utils/backend.server";
import { getItems } from "~/utils/backendIpsum.server";

const LIMIT = 10;
// hard coded for now:
const LAST_PAGE = Math.floor(500 / LIMIT) - 1;

type FetcherData = {
  items: Items;
};

export async function loader({ request }: LoaderFunctionArgs) {
  const url = new URL(request.url);
  const page = Number(url.searchParams.get("page")) || 0;

  const start = page * LIMIT;
  const items = await getItems({ start: start, limit: LIMIT });

  return json({ items });
}

export default function InfiniteIntersectionObserver() {
  const data = useLoaderData<typeof loader>();
  const fetcher = useFetcher<FetcherData>();
  const [items, setItems] = useState(data.items);
  const page = useRef(0);
  let loading = fetcher.state === "loading";

  const { ref, inView } = useInView({ threshold: 0 });

  // memoization isn't needed here
  // are there any potential issues with using it?
  const renderedItems = useMemo(() => {
    return items.map((item, index) => {
      if (index === items.length - 1) {
        return (
          <div key={item.id} ref={ref} data-id={item.id} className="px-3 py-6">
            {item.value}
          </div>
        );
      } else {
        return (
          <div key={item.id} data-id={item.id} className="px-3 py-6">
            {item.value}
          </div>
        );
      }
    });
  }, [items]);

  useEffect(() => {
    if (inView && fetcher.state === "idle" && page.current < LAST_PAGE) {
      page.current += 1;
      fetcher.load(`/infinite-intersectional-observer?page=${page.current}`);
    }
  }, [inView]);

  useEffect(() => {
    if (fetcher?.data && fetcher.data?.items) {
      const allItems = items.concat(fetcher.data.items);
      setItems(allItems);
    }
  }, [fetcher.data?.items]);

  return (
    <div className="max-w-screen-sm mx-auto">
      <h1 className="text-3xl">Infinite Intersectional Observer</h1>
      <p>
        Automatically load more data when a user scrolls to the bottom of the
        data's containing element.
      </p>
      <div className="relative">
        <div className="overflow-y-scroll max-h-96 mt-6 divide-y divide-slate-300">
          {renderedItems}
        </div>
        <div
          className={`${
            loading
              ? "transition-opacity opacity-100 duration-100"
              : "transition-opacity opacity-0 duration-100"
          } loading-message bg-sky-500 text-white text-center py-2 absolute bottom-0 left-0 right-4 rounded-sm`}
        >
          Loading
        </div>
      </div>
    </div>
  );
}

That seems like the way to go. It’s more efficient than attaching a handler to scroll events, and the code is easier to read 🙂

Infinite scrolling with a Link component