No, the despeeders do not weaken the tractor. The caution is because with lower gearing folks often try to pull or move more weight than the tractor should attempt. Since you have many lower gears than the typical 1st gear, it would "feel" like the tractor can pull a lot more than it would in regular 1st gear.
Unfortunately, as others have mentioned, your tractor with the deespeeders really limits it to a mowing machine only. The Lo Boy never did have cultivators that were made specifically for it, because it sits too low to the ground for belly mount implements (other than the mower deck). So you're out for cultivating unless you have pull-behind cultivators.
Now, the other problem with the despeeders is that they widen the track of the rear wheels, and space out the final drives away from their normal position. This is unfortunate because the final drives are the implement mounting points for all the rear-mounted implements that were made for a lo boy. Likewise, a "fast hitch" will also not mount on the tractor with the despeeders because the final drives are spaced out too far for it to bolt up to the tractor.
Yours is even more complicated because on the "highway mower" setup, they turned the final drives so the pans are facing backwards, instead of forward like on a standard lo-boy. They did this to lengthen the wheel base to make the tractor more stable for mowing on side slopes of highways, which worked very nicely. This configuration though, totally gets rid of all the mounting points to bolt up rear-mounted implements. So, yours will not pull a plow or disk, unless again you have aftermarket, drag-behind implements.
Having said all of this, your tractor would be PERFECT for one thing that many folks on here would like to do with a Cub. Rototilling!!! You're not likely to find a rototiller that will directly mount to a Cub, as the Howard Rotovator was the only one known to be built for a Cub, and they were only built for standard "high-boy" cubs, not for a lo-boy. However, if you search around Agri-Fab and Brinly both make gasoline powered roto tillers for mount on the back of garden tractors or 4-wheelers, and I'm sure that you could fabricate a mounting. The typical problem with a cub is the ground speed being too fast, but yours with the despeeders will solve that problem.
However, as others have said, your tractor is quite collectable, and if you would rather have a standard cub better suited for plowing, disking, cultivating, then I'm sure you could make a sweet deal and likely trade yours for a cub with all the implements. I'm sure you'll get many offers if you decide to go that route.