You can change the window size as follows.
▼Windows
You can exit full-screen mode by pressing Alt+Enter.
Hold down Shift when launching VRoid Studio to show window settings (see image).
▼Mac
If you hover the cursor over the top of the screen, the window bar will appear and you can press the green full-screen button at the top left to go back to window mode.
Hold down the Option key when launching VRoid Studio to show window settings (see image).