35 is pretty good. The game is designed for a fun experience with 30-50.
Also, keep in mind, raiding isn't implemented yet (on the to-do list). You can migrate to the neighboring settlement and attack them. You should have a decent fight with that setup. Just be aware that you can't migrate back to the previous tile just yet. That's another thing that's planned.