By swapping pulleys, I don't mean swapping the two that's there. Swap out either or both the stock pulleys for different ones. Pulleys can be bought at a variety of places, one is
https://phoenix-mfg.com/ Just be sure to NOT get a fractional horsepower pulley.
The rear pulley is typically larger, and a single pulley. Get one of the same shaft diameter and belt width. Half the diameter will result in twice the speed. That may not sound like much, but unless you upgrade the steering, might be enough. Depends on how tight your current setup is.
Remember, this may require a change of drive belt too. A smaller rear pulley would mean a shorter belt. It might very well require a modification of the tightening system, typically it's just an idler pulley connected to the clutch pedal. Depending on the idler/tightening set up, you might be able to modify it enough to take up enough slack to get away with the stock belt.
Every mower is different. Each has to be modified in it's own way.
Good luck, and happy motoring.
Dale