Loading some from config and some other from configSource

 After wasting way too much time on this, I came to the soultion:

<appSettings configSource="Settings.config">
  <add key="commonKey1" value="commonValue1" />
  <add key="commonKey2" value="commonValue2" />
  ... etc

is illegal, but the:

<appSettings file="Settings.config">
  <add key="commonKey1" value="commonValue1" />
  <add key="commonKey2" value="commonValue2" />
  ... etc

is perfectly fine... So it all came down to changing configSource attribute to file attribute. Now it works fine.