I have had this problem with a shorter run of pole - using regular wall plugs into the plaster/brickwork is seldom good enough - within a short time it all gets loose and dangerous...
I fix a planed wooden batten (in your case two lengths if it's 4m length) to the wall with Frame Fixings...
http://www.toolstation.com/images/library/stock/webbig/61889.jpg
Drill clearance holes in battens (10mm) - mark walls - Masonry drill walls (10mm) - bash in frame fixings with a hammer good and hard - decorate - screw pole brackets to the wooden batten with regular wood screws - fit pole - hang curtains - job done.
I would use a batten 20mmx75mm and frame fixings 10mmx120mm giving a good 4" hold into the brickwork.
With any luck you will be able to swing on the curtains like a wild jungle creature!
Good luck...
Peccavi
December 2009