if you wanna play 3D games in Windows 98, then they might not run very smoothly in Dosbox because there in no 3D hardware-acceleration by the GPU in Dosbox, its all done on the CPU (
with exceptions).
If you wanna achieve maximum 3D accelerated performance on old games that don't run on modern windows/GPUs out-of-the-box, you could first try to get them to run using a '
wrapper', which just translates old DirectX/Glide API calls to newer API calls that your modern GPU
does support.
However if the game relies heavily on Windows98 itself, then a wrapper will not work. If you still want to get the most performance then you could try
QEMU-3DFX, its the QEMU emulator with GPU passthrough, it runs a wrapper inside the VM that converts the API calls to OpenGL, that gets send to the hypervisor and processed there. But its hard to setup and the dev is toxic.
But QEMU-3DFX can be very instable because it doesn't try to mimic a real Win98 era CPU, so you might run into issues due to the CPU being too fast or CPU instructionset differences, so you would do the next step towards accuracy without sacrificing performance, and that is
SoftGPU, which is a driver for Type-2 hypervisors like Virtualbox/Vmware that gives you a properly hardware accelerated vGPU. Its slower then QEMU-3DFX, but its more stable.
If that doesn't work, you will have to look into more low-level emulation like x86box, PCEm or Dosbox-X, which will do graphics card emulation on the CPU so its slow, but very accurate and stable.
So for 3D games I always try it in this order Wrapper>QEMU-3DFX>SoftGPU>x86box/PCem/Dosbox-X>Buying a real retro PC.