Flaky Playwright tests

I decided to fix some of my flaky tests today, and it turns out that Playwright is too fast, so interaction with elements before hydration has finished can cause certain interactions to fail (https://playwright.dev/docs/navigations#hydration).

For example, one of the tests that I had to de-flake was a test for a login page, which has a username field and a password field. There were no unhandled exceptions, and the test seemed to just get stuck on the sign-in page. Modifying Playwright's config to retain videos for a failed run revealed that sometimes the username field would appear empty, but the password field always gets filled: the username input is an island that needs to be hydrated, and the password field isn't.

Waiting for an arbitrary amount of time is easy to do but never a good option because:

• Timing could change and cause a flake to resurface.
• It needs to be applied everywhere that renders the affected subtree, which can become a nightmare in a larger project.
• It could makes tests run substantially longer than they should, and having a lot of the add up in terms of both time and cost.

Setting the use.launchOptions.slowMo to 50 milliseconds in the Playwright config removed most of the flakes. Admittedly, I was really tempted to use "priorities" as an excuse and just leave it there but decided against that reasons similar to those above. 😓

According to the official recommendation:

The right fix for this issue is to make sure that all the interactive controls are disabled until after the hydration, when the page is fully functional.

It still needs to be applied on a case-by-case basis and I don't think there is a good way to catch these until a tests flakes, which isn't ideal.

I settled on a hook for now:

export function useDisabled(value: boolean): Signal<boolean | undefined> {
	const disabled = useSignal<boolean | undefined>(true);

	useEffect(() => {
		disabled.value = value;
	}, []);

	return disabled;
}

function SomeInteractiveElement(props: ComponentProps<"input">) {
    const disabled = useDisabled();

    return (
        <input {...props} disabled={disabled} /> 
    );
}