In Big Sur there is a new system setting under System Preferences / General / Prefer tabs. It controls whether related windows for an application should be presented in tabs.
The new tab support doesn't work well with Java apps. Open System Preferences, select General and then set Prefer Tabs to Never as in the screenshot.
This is the related Java bug: https://bugs.openjdk.java.net/browse/JDK-8256465