I wouldn't be so quick to rule out the trailer brakes depending on your brake controller.

How are the trailer brakes on your trailer setup? Do you use a separate controller?

Is the trailer able to be reversed when you on level ground or on a slight incline?