Hi, your approach seems to be fine. Interestingly I tried something similar and got relatively good results.

I would suggest following things to look into -

(1) you can play around with prompt, maybe change the prefix or try to provide a description of the task at the top!

(2) Passage and Answer could be long, you can try attention masking so that model only fine tunes to Questions and not to passage and answer.

Let me know if that helps, best of luck!