Esrolia, Caladraland, God Forgot-Esvular, and much of Maniria have conditions where you could grow olives.
In addition, in Esrolia-Caladraland, you can grow lemon fruits and oranges, along with grapes, and olives. Add in sassafras, plenty of spices, chili peppers, oregano, and sugar cane.
Olives would fall within the ambit of the local Earth goddess or Grain Goddess. Ernalda, Esrola, etc. combined with Flamal. Bless Crops and Bear Fruit are your friends.