Yes, -mabi=lp64 would be the correct option for 64-bit soft-float code.
The linker error indicates that you are trying to link hard-float libraries with soft-float libraries. Why you got this depends on exactly how you built the toolchain. If you built a hard-float toolchain, and did not enable multilibs, then you would have built only one set of libraries which are hard-float, and which will not be link compatible with soft-float code.
You can fix that by configuring a toolchain to emit soft-float code directly, e.g. make -march=rv64i -mabi=lp64 the default. You can do this by specifying --with-arch=rv64i --with-abi=lp64 when configuring gcc.
Alternatively, you can build multiple copies of the libraries, for various combinations of arch/abi. You can do this by specifying --enable-multilib when configuring gcc. You can see the list of libraries supported by gcc by using “gcc --print-multi-lib”.