It sounds like it's definitely a case of "the pot comes first". If you're in for a fun friendship or easy-going relationship that's one thing, but if you want anything more serious, I'd tread carefully. Someone who blows off plans for a weekend of stoning with roomies is not usually someone you'd want to get really involved with - for one thing, it's really boring!! !
If you do bring up the fact that you sometimes feel a bit blown off by him, for god's sake don't mention the pot - that'd be a sure-fire way to get him defensive and angry. Keeping it non-specific, by saying how disappointed you are how often your plans to get together fall through, lets him know that you've noticed him getting distracted and bunking off, but simultaneously gives him a bit of a compliment by letting him know that you'd really like to see him more.
But geez, I'd still be careful if he's fobbing off to get stoned all weekend!