SWRA682 December 2020 CC1310 , CC1312PSIP , CC1312R , CC1314R10 , CC1350 , CC1352P , CC1352P7 , CC1352R , CC1354P10 , CC1354R10
Theoretically, there is an optimum separation/symbol rate setting if you simultaneously minimize the receiver filter bandwidth. Every halving of receiver filter bandwidth improves sensitivity by 3 dB whereas sensitivity vs separation/datarate decreases with about 1.5-2.5 dB per halving down to a certain limit where the loss increases very fast. In our experience a modulation index (h =2 x deviation/symbol rate = separation/symbol rate ) = 0.5 - 1 is a good design compromise.
For 4-GFSK the modulation index for inner symbols should be >= 1/3. Lower modulation index will degrade sensitivity significantly.