Yes, it will make a difference if the exhaust valve is set incorrectly. This engine has an internal compression release for starting (engine turning below 500 rpm) and the valves have to be set correctly for it to work. If you set the exhaust at .007", and were a little off, the clearance may be large enough to defeat the compression release. Try setting the exhaust clearance to .005".
I test the starter with a fully-charged battery and a set of jumper cables. Connect the positive terminal of the battery directly to the starter power lug, then connect the negative terminal to the tractor chassis or engine block. It might be best to first connect one negative lead to the tractor, then connect the other end to the battery negative terminal, because there will be some amount of arcing when the final connection is made. Testing this way eliminates the ignition switch, the solenoid, and all wiring connections as the source of the problem.
BTW: a fully-charged battery will read 12.7 volts, a reading of 12 volts indicates a 50% charged battery. At 11.5 volts you only have about 10% charge. Letting a battery get below 11 volts greatly decreases battery life.